Downloaded By: [University of Sydney] At: 00:18 27 September 2007
Medical Informatics and the Internet in Medicine September 2007; 32(3): 225 – 240
Are we measuring the right end-points? Variables that affect the impact of computerised decision support on patient outcomes: A systematic review
VITALI SINTCHENKO1,2, FARAH MAGRABI1 & STEVEN TIPPER1 1 2
Centre for Health Informatics, University of New South Wales, Sydney, Australia and Western Clinical School, University of Sydney, Sydney, Australia
(Received for publication 6 January 2007; accepted 11 May 2007)
Abstract Previous reviews of electronic decision-support systems (EDSS) have often treated them as a single category, and factors that may modify their effectiveness of EDSS have not been examined. The objective was to summarise the evidence associating the use of computerised decision support and improved patient outcomes. PubMed/Medline and the Database of Abstracts were searched for randomised controlled trials (RCT) of EDSS from 1 January 1994 to 31 January 2006. Twenty-four RCT studies from 97 reviewed were selected, eight of them examined systems supporting decisions for patients who presented with an acute illness, and 16 studies enrolled patients with chronic conditions. Overall, 13 (54%) of the studies showed a positive result, and 11 (46%) were with no impact. Critiquing and consultative systems showed the impact in 71% and 47% of studies, respectively. All systems targeting decisions related to acute disease improved patient outcomes compared with 38% of systems focused on the management of chronic conditions (P ¼ 0.005). Provision of EDSS improves prescribing practices and treatment outcomes of patients with acute illnesses; however, EDSS were less effective in primary care. Complex interventions as clinical EDSS may require new metrics of assessment to describe the impact on patient outcomes.
Keywords: Decision-support systems, clinical decisions, clinical outcomes, systematic review
1. Introduction During the past three decades, the impact of electronic decision support systems (EDSS) on health care has been examined on many occasions [1 – 11]. However, the majority of evaluations of EDSS typically measured their effect on the process of care rather than on patient outcomes, and their results have not been concordant [1,12]. To respond to the variability in clinical settings and the quality of studies reviewed, and to resolve the controversy around the clinical effectiveness of EDSS, Kaplan [8] called for better-quality research evidence.
Correspondence: V. Sintchenko, Centre for Health Informatics, University of New South Wales, UNSW Sydney 2052, Australia. Fax: þ612-9385-9006. E-mail:
[email protected] ISSN 1463-9238 print/ISSN 1464-5238 online Ó 2007 Informa UK Ltd. DOI: 10.1080/14639230701447701
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
226
V. Sintchenko et al.
Randomised controlled trials (RCT) have been suggested as the best experimental approach to examine the impact of computerised decision aids on health care [1,8]. Yet the number of RCT focused on such assessment remains small. A systematic review that analysed 28 RCT from 1983 to 1992 concluded that patient outcomes were unchanged despite increased compliance with guidelines [1]. Another systematic review of computerised clinician-focused interventions, which analysed 25 studies from 1992 to 1998, confirmed the positive effect of these tools on performance measures but was inconclusive on patient outcomes due to lack of data [5]. Hunt and colleagues, in a most clinically relevant systematic review, found a benefit in only six of 14 controlled trials assessing patient outcomes but stated that, of the remaining eight studies, only three had a power of greater than 80% to detect a clinically important effect [4]. These observations have been confirmed more recently, and the need for new approaches to assess the impact of these tools on medical practice and quality of care has been highlighted [8,10]. However, this call has been largely ignored, and recent updates of earlier reviews [11,13] still concluded that effects of computerised decision support on clinician performance and patient outcomes remain understudied and, when studied, inconsistent. This is, perhaps, not surprising, as the nature of clinical decisions supported by specific systems and the relationship between clinical decisions and patient outcomes has received little attention so far [14]. Despite the variability in interventions and clinical setting, EDSS has often been treated as a single homogenous category, and factors that may modify the effectiveness of EDSS have not been systematically examined [10]. Evidence suggests that clinical effectiveness of EDSS may be affected by the type of clinical decision support [15,16] and the severity of patient presentation [14]. Evidence suggests that clinical outcomes are primarily influenced by the severity of a patient’s illness and the quality of clinical decisions [17 – 19]. Therefore, we hypothesised that the magnitude of the EDSS impact on patient outcomes is affected by the clinical decision task, decision setting (acute or chronic care), and type of decision support, and reviewed the current evidence to test these associations. This systematic review focuses on clinical studies that have evaluated the effects of computerised decision support on patient outcomes and attempted to study the impact of the type of clinical decisions and decision-support systems as well as the severity of patient presentation on the effectiveness of EDSS use.
2. Methods 2.1. Study identification PubMed/Medline and the Database of Abstracts and Reviews (DARE) were searched for English-language publications from 1 January 1994 to 31 January 2006 using the following combination of medical subject headings (MeSH), text words, and publication types: (‘outcome’ or ‘outcomes’) and (‘decision support system’ or ‘situation assessment tool’ or ‘computerised decision support’ or ‘expert system’) or (‘health technology’ or ‘computer-assisted diagnosis’ or ‘computer-assisted patient management’ or ‘electronic prescribing’ or ‘electronic test ordering’ or ‘artificial intelligence’ or ‘mobile computing’). The search was carried out in February 2006. The reference lists of the articles selected for inclusion were also reviewed. 2.2. Study selection Our inclusion criteria were: prospective study in a clinical setting; participants were health professionals in a clinical practice; assessment included patient outcomes and
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
Computerised decision support on patient outcomes
227
clinical performance (a measure of process of care); prospective collection of data and a control group so that patient care using an EDSS could be compared with care without one. We designed the selection criteria to limit the analysis to well-defined clinical tasks and outcomes. Any system that provided electronic patient-specific information to support clinical decisions was considered an EDSS for the purposes of this review. Randomised control trials (RCT), where the allocation of patients, encounters, practitioners, or practices to the EDSS intervention was stated as being randomised, were included. Duplicate reports of studies were eliminated by comparing authors’ names and the type and location of the study. Reports other than in English were not included. No criteria with respect to trial methodology were defined. Studies where clinical outcomes were measured or assessed by patients were excluded. 2.3. Data extraction and study evaluation Full-text articles were retrieved for all titles considered for detailed review. All studies with positive and negative impacts were included. Because of the small number of articles and the heterogeneity of study settings and systems described, including clinical problems, clinician and patient selection, methods of intervention, and measures of outcomes, meta-analysis of the trial results was not considered possible. Thus, measures of clinical performance and patient outcomes were characterised and classified for each study according to whether a significant effect was reported. Proportions of positive studies in a given category were calculated. To be considered positive, an impact in a study needed to show a statistically significant change in patient outcome or a targeted clinical practice (either an increase in intended activities or a reduction of ineffective actions). Clinical decision tasks supported by respective EDSS were classified into diagnosis, treatment, or plan [20], and the system functionality was identified as consultative (e.g. provision of evidence-based guidelines, assistance in patient’s assessment, drug dosing, etc.) or critiquing (e.g. alerts or reminders) [21]. Systems were divided into ones that support clinical decisions relating to the management of patients with acute and often severe illness or chronic conditions [22,23]. Severity of disease referred to the severity and importance of a particular diagnosis (often the principal diagnosis) to the patient’s risk of an untoward outcome, regardless of the patient’s other health conditions. For example, a patient with a principal diagnosis of cancer may be staged from early less severe stages (representing local disease) and to the most severe (representing widely metastatic spread). Severity of disease is often diagnosis-specific, represents the acuity of presentation at the time of EDSS use, and measures the importance of the diagnosis. Each study included was independently evaluated and characterised by three co-authors and with individual variations reviewed and resolved by group consensus. 2.4. Types of outcome measures Studies were included if there were analysable data for any objective measure of patient outcomes (morbidity or mortality including adverse drug events (ADE))—Level 1 outcome— or surrogate outcomes (e.g. observed errors, intermediate outcomes with a well-established connection to the clinical outcomes of interest) or other measurable variables with an indirect or unestablished connection to the target clinical outcome (eg, compliance with a ‘recommended’ practice)—Level 2 outcome (Table I) [24].
228
V. Sintchenko et al.
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
Table I. Hierarchy of clinical outcome measuresa. Level of outcome
Outcome measure
Description
1 2
Clinical outcome Surrogate outcome
3
None
Any measure of morbidity or mortality including adverse events Observed errors, compliance with evidence-based clinical protocols, intermediate outcomes (e.g., laboratory test results) with a well-established connection to the clinical outcomes of interest No outcomes relevant to implementation of electronic decision support (e.g., the study describes an application of decision support system but reports no measured outcomes)
a
Adapted from [24].
3. Results 3.1. Study settings Thirty-two articles were identified at the end of the selection process (Figure 1). A small proportion of the studies retrieved from the secondary bibliography (12/4,223 or 0.28%) confirmed the effectiveness of the search terms. Eight out of the 30 papers identified were excluded after careful analysis because of insufficient data on clinical outcomes (n ¼ 6) or because clinical outcomes were assessed subjectively only by patients (n ¼ 2). Selected studies are presented in Tables II and III. The 24 RCT included in this review were carried out in five different countries: 10 in the USA, seven in the UK, one in Canada, and one in Norway. They varied in size from 175 patients [33] to nearly 18,000 patients enrolled [32]. Evidence suggests a shift of interest and resources over time from hospital-based trials of decision support systems to those in a primary care setting. What follows is an analysis of themes and trends in the literature reviewed. 3.2. Decision tasks Eight controlled trials studied systems supporting clinical decisions for patients who presented with an acute illness or severe exacerbation of chronic disease in hospital setting (Table II), and 16 studies enrolled patients presenting to primary care practitioners with severe chronic diseases (Table III). The studies assessed the impact of EDSS on a variety of clinical decisions including treatment decisions (11 studies), diagnostic decisions (four studies) and planning decisions (nine studies). Treatment decision studies targeted urgent antibiotic prescribing for hospitalised patients with severe infections [26,30,31,32] or for outpatients in primary care [44,48], drug-dosing of anticoagulants [25,38], and management of diabetes [27,33 – 36], as well as therapy for cardiovascular events and asthma [28,41,45]. For the papers dealing with diagnosis or planing tasks, the evaluated systems supported diagnoses and management of asthma [40,41,48], chronic heart disease and hypertension [37,39,41,44,45,48], treatment of diabetes [27,33 – 37], and depression [42,46], as well as diagnostic decisions relating to the care for patients with respiratory failure [29]. 3.3. Forms of decision support The majority of treatment decision-support systems provided alerts and reminders for prescribing decisions (Tables II and III). EDSS assisted with drug-dosing of agents with a
229
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
Computerised decision support on patient outcomes
Figure 1. Selection process of studies for systematic review.
high risk of ADE [25,30,31,38] or generated alerts during provider order entry [43,47]. One system [45] assisted clinicians in the treatment of patients with stroke by generating estimates of vascular events. Diagnostic decisions were supported by systems designed to provide point-of-care access to evidence-based guidelines [35,40,41,44,48], focus attention on specific investigations required to manage patients with depression [42,46] and respiratory distress [29], or assess the risk of complications of chronic conditions [39,45].
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
230
V. Sintchenko et al.
Of the RCTs we reviewed, nine systems were classified as consultative EDSS [25 – 27,31,38,40,43,45,46,48]. Four systems critiqued prescribing tasks [28,30,32,47], and three aided diagnostic tasks by providing alerts and reminders [33,34,42] (Table IV). Eight systems supported management of patients with chronic conditions [35 – 37,39,41,44,47,48], and only one supported those with acute illness [29]. 3.4. Impact on patient outcomes Outcome measures were reported in all studies, ranging from mortality and length of stay to the rates of optimal decision-making consistent with evidence-based practice (Tables II and III). Only eight studies (33%) analysed the impact of EDSS on Level 1 clinical outcomes [27,29 – 31,37,38,42,46]. Overall, 13 (54%) of the studies showed a positive result, and 11 (46%) were negative (Table IV). Of the positive studies, only one showed an improvement in Level 1 clinical outcomes [31] with a majority of studies (50% or 12/24) demonstrating an improvement only in surrogate outcomes and other variables (Level 2 outcomes) as indirect measures of clinical outcomes (P ¼ 0.04). Critiquing and consultative systems demonstrated positive impact in 83% and 50% of controlled studies, respectively (Table IV). Furthermore, only one consultative system which supported drug dosing as a part of provider order entry showed an improvement in patient length of stay (Level 1 outcome) along with a decrease of inappropriate dosing and frequency (Level 2 outcomes) [31]. All systems targeting clinical decisions related to acute disease or acute exacerbation of chronic disease improved patient outcomes compared with 38% of systems focused on the management and treatment of chronic conditions (P ¼ 0.005) (Table V). For example, no benefit with respect to the management of asthma, angina, or major depression was observed [40 – 42,44,48]. Two of five RCT targeting decisions related to diabetes reported significant changes in compliance with practice guidelines [34,36]. In both studies, the EDSS was a part of integrated care interventions to improve evidence-based management of chronic conditions.
4. Discussion 4.1. Determinants of EDSS effectiveness More than half of the trials identified in our search showed a clinical benefit, and no study found the use of a decision-support system to be detrimental. No RCT studies exploring applications of the general medical diagnosis systems were identified in our search. Reviewed evidence suggested that the effectiveness of EDSS is dependent on or can be predicted by the severity of patient presentation, type of clinical decisions, and type of decision support. It appears that EDSS were more effective in acute care than when less wellstructured chronic care decisions were targeted by decision support. Considering the setting of care, all eight (100%) in-patient studies were positive, compared with only five out of 16 (31%) primary care studies (Table IV). These results may be confounded by the fact that many in-patient studies were concerned with prescribing activities which may be easier to optimise than some other clinical activities [49]. At the same time, these findings probably reflect the higher impact of clinical decisions in acute care on patient outcomes. Our findings also confirmed previous observations that electronic decision support has the potential to improve the quality of clinical decisions, patients’ outcomes, and safety [5,16]. However, the effectiveness of EDSS is not uniform. Controlled trials reviewed here strongly
Treatment
Plan
Treatment
451
2,181
No change in LOS, significant increase in protocol compliance (P 5 .0001) 719
17,828
6,371
Chertow, 2001, US [31]
Dexter, 2001, US [32]
Patient-days reported only.
a
Treatment
22,509a
Kuperman, 1999, US [30]
Treatment
Treatment
Diagnosis
200
East, 1999, US [29]
Rossi, 1997, US [28]
Treatment
575
Poller, 1993, UK [25] Evans, 1994, US [26] Overhage, 1997, US [27] Adherence to guidelines (Level 2)
Decision task
No. of patients
First author, year, country
Critiquing—Reminders during provider order entry about preventive measures
Consultative—Drug-dosing support during order-entry
Critiquing—Alerting for critical events involving laboratory results and medications
Consultative—Support for mechanical ventilation of patients with ARDS
Critiquing—Computer generated reminders for the treatment of hypertension
Consultative—DSS to assist warfarin control Consultative—Antibiotic selection consultant Consultative—Computerised suggestions for corollary orders for selected tests and treatment
EDSS function
Rates of preventive therapies or vaccination (Level 2)
Number of admissions, laboratory test ordered, blood pressure measurement, compliance with guidelines (Level 2) LOS, mortality (Level 1), morbidity by lung dysfunction scores or iatrogenic lung injury (Level 2) ADE (Level 1), interval from when a critical result is available for review until an appropriate treatment is ordered (Level 2) LOS (Level 1), rates of appropriate prescriptions by dose and frequency (Level 2)
Rate of appropriate anticoagulation therapy (Level 2) Rate of appropriate antimicrobial therapy (Level 2) Average LOS (Level 1)
Outcome measures
Table II. Cohort characteristics of studies from hospital setting/acute illness management.
No change in the rate of ADE, 38% decrease in time until an appropriate treatment is ordered (P ¼ .003) LOS decreased to 4.5 from 4.3 days (P ¼ .009), 13% decrease in inappropriate dose (P 5 .001) and 24% decrease in inappropriate frequency (P 5 .001) Increase in vaccination rates (P 5 .001)
No change in LOS or mortality but significant reduction in morbidity
Increased compliance with guidelines, no impact on other measures
Improved dosing at a higher density of anticoagulation (P ¼ .044) 17% increase (P 5 .001) in appropriate therapy
Effect on patient outcomes
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
Computerised decision support on patient outcomes 231
274
164
357
DICET, 1994, UK [34]
Nilasena, 1995, US [35]
Lobach, 1997, US [36] Hetlevik, 2000, Norway [37]
200
Rollman, 2002, US [42]
447
McCowan, 2001, UK [40]
4001
614
Montgomery, 2000, UK [39]
Eccles, 2002, UK [41]
224
Fitzmaurice, 2000, UK [38]
1034
175
No. of patients
Mazzuca, 1990, US [33]
First author, year, country
Diagnosis
Treatment
Diagnosis
Diagnosis
Treatment
Treatment
Plan
Plan
Plan
Plan
Decision task
Critiquing—Computer generated reminders on management of depression
Consultative—Computerised guidelines for management of asthma and angina
Consultative—DSS for management of asthma
Consultative—DSS for assessment of cardiovascular risk and management of hypertension
Critiquing—Computer reminders to consider a recommendation for a diabetic patients Critiquing—Computer reminders to consider a recommendation for a diabetic patients Consultative—Computerised preventive care guidelines for diabetes Consultative—Computerised guidelines for diabetes Consultative—Computerised guidelines for diagnosis of hypertension and diabetes Consultative—DSS to control anticoagulant dose
EDSS function
Consultation rates, adherence to evidence-based recommendations and patient-reported outcomes (Level 2) Depression score at 3 and 6 months (Level 1)
Markers of diabetes, blood pressure (Level 1), adherence to protocol (Level 2) Mortality (Level 1), ADE based on biochemical surrogate measures (Level 2), therapeutic control of anticoagulation (Level 2) Proportion of patients identified for high cardiovascular risk, blood pressure and prescribing of cardiovascular medications (Level 2) Medication use, rate of exacerbations (Level 2)
Protocol compliance (Level 2)
Adherence to guidelines (Level 2)
Routine care visits, glycated haemoglobin assessments (Level 2)
Protocol compliance (Level 2)
Outcome measures
Table III. Cohort characteristics of studies from primary care setting/chronic illness management.
No change
No change
9% decrease in asthma exacerbations
No change
No change
No change
(continued)
50% increase (P ¼ 0.05)
No change
Significant increase in protocol compliance (P 5 0.05)
No change
Effect on patient outcomes
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
232 V. Sintchenko et al.
706
Tierney, 2003, US [44]
762
235
706
Thomas, 2004, UK [46]
Lester, 2005, US [47]
Tierney, 2005, US [48]
1,952
12,560
Tamblyn, 2003, Canada [43]
Weir, 2003, UK [45]
No. of patients
First author, year, country
Plan
Plan
Plan
Treatment
Plan
Treatment—Prescribing
Decision task
Consultative—Computerised guideline for managing asthma and chronic obstructive pulmonary disease
Consultative—DSS for stroke management (generated estimates of vascular events) Consultative—Computerised psychosocial assessment linked to guidelines Critiquing—Computerised reminders to consider statin prescription
Consultative—Computerised guideline for managing heart diseases
Consultative—Prescribing support during provider order entry
EDSS function
Table III. (Continued).
Adherence to recommendations, emergency department visits, hospitalisations (Level 2)
General Health Questionnaire score at 6 weeks and 6 months after treatment (Level 1) LDL levels, protocol compliance, change in hyperlipidaemia prescriptions (Level 2)
Adherence to recommendations, acute exacerbations of heart disease, medication compliance (Level 2) Rate of optimal antithrombotic therapy (Level 2)
Rates of appropriate prescriptions (Level 2)
Outcome measures
Decrease in post-intervention LDL levels (P ¼ 0.04), 7 times increase in compliance (P 5 0.001) No change
No change
No change
13% increase in appropriate prescribing and vaccination rates (P 5 .01) No change
Effect on patient outcomes
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
Computerised decision support on patient outcomes 233
234
V. Sintchenko et al.
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
Table IV. EDSS functions and their impact on clinical outcomes. Improvement in outcomes, percentage of positive studies (no.) EDSS function
Level 1 outcomes
Level 2 outcomes
25% (1/4) 0% (0/3) 14% (1/7)
50% (8/16) 83% (5/6) 59% (13/22)a
Consultative Critiquing Total a
P ¼ 0.038 between Level 1 and 2 outcomes. Table V. Severity of presentation (decision domain) and type of decision support and outcomes of interventions. Significant improvement in patient outcomes, percentage of positive studies (no./total number of studies with outcome measured)
Decision domain and type of clinical decision Acute disease or exacerbation of chronic disease Plan Diagnosis Treatment Chronic disease Plan Diagnosis Treatment
Level 1 outcomes 25% 0% 0% 50% 0% 0% 0% 0%
(1/4) (0/1) (0/1) (1/2) (0/3) (0/1) (0/1) (0/1)
Level 2 outcomes 100% 100% 100% 100% 38% 43% 50% 25%
(8/8) (1/1) (1/1) (6/6) (5/13)a (3/7) (1/2) (1/4)
a
P ¼ 0.005 between acute and chronic disease EDSS.
indicate that process measures, rather than ‘hard’ clinical outcomes, are more often accepted for evaluation of EDSS interventions. For example, the most common improvement in practice observed was the increase in adherence to clinical guidelines and protocols. The magnitude of the improvement in compliance ranged from 13% to 17% in a majority of successful interventions to a 227% increase in overall compliance with recommended diabetes care procedures in the prompted group of physicians in one study [27]. This observation supports classification of EDSS as a heterogenous group, in which each system has a unique efficacy profile. The EDSS effectiveness appeared to vary across RCT studies targeting different clinical tasks (Tables IV and V). Factors influencing success or failure of health informatics systems were reviewed elsewhere [50]. However, the role of decision tasks and different EDSS types on the impact of clinical EDSS has received little attention. Electronic decision aids demonstrated modest effects in clinical trials with chronically ill patients where the relationship between clinical decision and outcome is less predictable. Critiquing EDSS have worked better, providing reminders for preventive care or assisting with drug prescribing. The usefulness of EDSS has likely arisen from their ability to use clinical information to answer straightforward questions, such as whether a certain drug was contraindicated or what was an appropriate dose for the medication [6,12]. Implementation of EDSS in these areas should be a high priority [23]. Consultative decision aids supporting the management of chronic illness in primary care showed a modest impact on the clinical outcomes measured. None of the RCT involving these EDSS demonstrated any significant improvement in Level 1 outcomes. It appears that clinical EDSS have a lesser effect on patients with chronic conditions than patient-level
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
Computerised decision support on patient outcomes
235
interventions such as computer-assisted insulin dose adjustment or utilisation of home glucose records for diabetes sufferers [51,52]. The majority of successful trials in our review provided computerised guidelines to medical practitioners or nurses, or were designed as administrative aids for registration and recall. Lack of evidence of the effectiveness of EDSS in chronic care may be partially related to the low risk of poor patient outcomes in both control and test groups [16]. This is of concern, especially in view of the current shift of focus to chronic care practice improvements [22] and the fact that the majority of recently published RCTs (Table III) tested the latter systems. It is clear that the major factor limiting sustainable impact of EDSS on clinical practice is lack of knowledge of clinicians’ information processing, information needs and evidence uptake [12,15,53 – 55]. The absence of a significant effect on clinical outcomes may reflect problems with the integration of systems within the clinical decision process [56] or the level of EDSS adoption rather than unsatisfactory performance of a particular system itself [53,57]. Yet, this issue has not received appropriate recognition in reported studies. Only five trials reported on the adoption of EDSS, yet four of them documented low rates of the system usage [29,37,41,44,45]. The majority of them studied electronic evidencebased guidelines for chronic conditions in a primary care setting where they may have been regarded as optional rather than standard. For example, computerised guidelines for diagnosis of hypertension and diabetes were used only in 12% of patients with diabetes in one controlled Norwegian study [37], while 69% of patients did not receive the optimal therapy suggested by EDSS in another trial held in the UK [45]. It is plausible that the low uptake of EDSS in primary care studies significantly affected the negative findings of respective trials. Previous research suggests that compliance with treatment regimens for acute illness has been estimated at only 30 – 40%, despite the clearer relationship between clinical decision and outcome [58]. When this relationship is even less evident, as in chronic conditions, a high uptake of clinical protocols is less likely [22]. It would be consistent with current evidence that EDSS use, and acceptance by health-care practitioners remains low [57,59,60]. 4.2. Are we measuring the right thing? The second theme is the potential lack of sensitivity of current approaches to outcome measurement. First, lack of justification for the choice of clinical outcomes measured in trials is of concern. Most of the included studies primarily measured Level 2 clinical outcomes, because Level 1 outcomes are less frequent. Yet almost half of the RCT included in our review failed to demonstrate a significant change in any primary or secondary clinical endpoints. Second, the duration of follow-up optimal for the detection of potential EDSS effects remains largely unknown. Patients in the studies reviewed here were followed for 3 – 12 months, and so could not exclude the possibility of any delayed effects. It is possible that larger, longer, and costlier trials may be required to estimate the effects of EDSS interventions on clinical outcomes, especially in a primary care setting [61]. It seems that morbidity, mortality, length of stay, and other Level 1 outcomes are not optimal for assessment of interventions to improve chronic care decisions, as they are unlikely to be altered by such interventions in many cases, because there is less opportunity for change. Many other outcome measures, such as resolution of symptoms and signs, consultation rates, etc., used in EDSS studies, are prone to bias or are likely to be influenced by factors unrelated to the quality of clinical decision-making [62]. The choice of outcome measures depends on the stated goals of a particular study. The choice of not only quantitative measures of outcomes, but also some qualitative measures
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
236
V. Sintchenko et al.
[59], which may result in clinically as well as statistically significant improvements, is warranted. For complex problems of diagnosis and management, condition-specific outcomes, such as diagnosis of abdominal pain or choice of antibiotic agents, would be appropriate [20,63]. However, justification and full data on outcomes used to measure the impact of EDSS are often lacking in published studies. For example, the study of Poller et al., which compared three systems with traditional dosing showed that computer dosing was comparable with traditional dosing at a lower intensity of coagulation but performed significantly better at a higher intensity [25]. No information was reported, however, regarding patient outcomes in terms of haemorrhagic or thrombotic episodes. Certainly, no single study can explore all of the potential areas in which medical EDSS may be expected to have a significant impact. Nonetheless, it would be appropriate to investigate the effects EDSS may have on patient outcomes, should these systems be widely introduced in clinical practice. The outcome indicator of choice for EDSS assessment should have the statistical power to detect differences in quality of decision-making. Formal theories of measurement offer a range of nominal, ordinal, interval, or ratio scales, including discrete and continuous and bounded or unbounded variables to quantify the change in a parameter and emphasize that these variables should be integrated into the theoretical structure describing measurement [64]. If various scores of measurement are combined with little thought about information content or how they are related, such measurement may be of uncertain relevance and at best sub-optimal or at worst misleading. For example, it has been shown that mortality is an insensitive measure of the quality of management of community-acquired pneumonia or acute myocardial infarction [18,65]. Furthermore, mortality and patient length of stay are affected by institutional factors such as the size of the hospital, the ‘bed-pressure’, nurse – bed ratios, and clinicians’ practice style. Such factors can partly explain the documented variation in length of stay between hospitals without variation in outcomes [66]. This is not always the case for commonly used EDSS outcome measures. The most important conclusion that can be drawn is that such a complex intervention as clinical EDSS may require new or different metrics of assessment to be able to fully describe the impact of the system under study on clinical decisions and patient outcomes. By measuring the right thing, we mean measuring a variable that constitutes a well-chosen compromise between finality and responsiveness or sensitivity to changing professional conditions. The main arguments for this are (a) the EDSS interventions are aimed at the health-care practitioner but outcome measures are patient-based; (b) the intervention is indirect, in the sense that in itself it does not influence disease activity; and (c) the effects on outcomes are expected to be relatively small, and so the outcome measures used to date may not be sensitive enough to detect small but clinically relevant changes. It is possible that the majority of RCT performed on the effectiveness of EDSS in primary care, mainly in patients with hypertension, diabetes, or anticoagulation need, failed to demonstrate a significant improvement in patient outcomes for those very reasons. However, we are not convinced that statistical means are the best way to demonstrate benefits of EDSS. Systematic theory seems to lag behind knowledge based on practice, and there is no framework of quantitative attributes to measure the ‘value’ of outcomes nor a validated scaling system to quantify clinically relevant outcomes. 4.3. Study limitations Results of this study are based on the analysis of the small number of available RCTs recently published in English literature with reported patient outcomes; the publication bias cannot be
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
Computerised decision support on patient outcomes
237
ruled out. It is also possible that we underestimated the real effectiveness of EDSS by excluding published trials that did not reported data on clinical outcomes. However, the fact that seven (44%) of the identified studies were negative confirms the view that negative studies are considered worthy of publication and that the publication bias should not alter our conclusions significantly [67]. The heterogeneity of the studies weakened any general inferences and prevented a meta-analysis, which could have allowed a more quantitative assessment [67,68]. Evaluation studies should be explicit about determinants of external validity such as the uptake of a system, EDSS run-in period, implementation strategies, differences between the trial protocol and routine practice, uniformity of patient populations, and risks of poor outcomes in the intervention and control groups. Because of the resource-intensive nature and logistical complexity of RCT, there is a call for a broader investigative approach to address the lack of evidence on clinical effectiveness of EDDS and a suggestion to employ interrupted time series when intervention is tested repeatedly both before and after EDSS use as a more practical alternative study design allowing detection of many confounding variables [10]. 5. Conclusions Published randomised control trials of EDSS indicate that implemented systems were more effective in a hospital setting than when chronic care decisions were targeted by computerised decision support. The type of decision task and functionality of clinical decision support are likely to influence the impact of the system on patient outcomes. High-level evidence confirms that provision of computerised decision support improves prescribing practices and treatment outcomes of patients with acute illnesses; however, it appears that EDSS were generally less effective in changing doctors’ performance or health outcomes in primary care. Such information is essential to identify types of patients and clinical decisions that will benefit most from EDSS. Further research is needed to quantify the range of benefits of EDSS and explore new measurement metrics to enable detection of clinically significant changes in patient outcomes and to enhance the appropriate clinical use of electronic decision support. Evaluation studies should be explicit about determinants of external validity. Acknowledgements The authors thank Enrico Coiera for his comments on the earlier version of the paper. VS was supported by a fellowship from the National Institute of Clinical Studies, Melbourne, Australia.
References 1. Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcomes. A critical appraisal of research. Annals of Internal Medicine 1994;120:135 – 142. 2. Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computerbased clinical reminder systems for preventive care in the ambulatory setting. Journal of the American Medical Informatics Association 1996;3:399 – 409. 3. Fitzmaurice DA, Hobbs FDR, Delaney BC, Wilson S, McManus R. Review of computerised decision support systems for oral anticoagulation management. British Journal of Haematology 1998;102:907 – 909. 4. Hunt DL, Haynes B, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes. A systematic review. Journal of the American Medical Association 1998;280:1339 – 1346.
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
238
V. Sintchenko et al.
5. Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: A systematic review of functionality and effectiveness. Journal of the American Medical Informatics Association 1999;6:104 – 114. 6. Walton R, Dovey S, Harvey E, Freemantle N. Computer support for determining drug dose: systematic review and meta-analysis. British Medical Journal 1999;318:984 – 990. 7. Estabrooks C, Goel V, Thiel E, Pinfold P, Sawka C, Williams I. Decision aids: are they worth it? A systematic review. Journal of Health Services Research and Policy 2001;6:170 – 182. 8. Kaplan B. Evaluating informatics applications – clinical decision support systems literature review. International Journal of Medical Informatics 2001;64:15 – 37. 9. Bennett JW, Glasziou PP. Computerised reminders and feedback in medication management: a systematic review of randomised controlled trials. Medical Journal of Australia 2003;178:217 – 222. 10. Delpierre C, Cuzin L, Fillaux J, Alvarez M, Massip P, Lang T. A systematic review of computer-based patient record systems and quality of care: more randomised clinical trials or a broader approach? International Journal for Quality in Health Care 2004;16:407 – 416. 11. Garg AX, Adhikari NKJ, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, Sam J, Haynes RB. Effect of computerised clinical decision support systems on practitioner performance and patient outcomes. A systematic review. Journal of the American Medical Association 2005;293:1223 – 1238. 12. Sim I, Gorman P, Greenes RA, Haynes RB, Kaplan B, Lehmann H, Tang PC. Clinical decision support systems for the practice of evidence-based medicine. Journal of the American Medical Informatics Association 2001;8:527 – 534. 13. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. British Medical Journal 2005;330:765 – 712. 14. Kushniruk AW. Analysis of complex decision-making processes in health care: Cognitive approaches to health informatics. Journal of Biomedical Informatics 2001;34:365 – 376. 15. Wyatt JC. Clinical questions and information needs. Journal of the Royal Society of Medicine 2000;93:168 – 171. 16. Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T. Reducing the frequency of errors in medicine using information technology. Journal of the American Medical Informatics Association 2001;8:299 – 308. 17. Kost GJ. Handbook of clinical automation, robotics, and optimization. New York: Wiley; 1996. 18. Mant J, Hicks N. Detecting differences in quality of care: sensitivity of measures of process and outcome in treating acute myocardial infarction. British Medical Journal 1995;311:793 – 796. 19. Jefford M, Stockler MR, Tattersall MHN. Outcome research: what is it and why does it matter? Internal Medicine Journal 2002;33:110 – 118. 20. Fox J, Thomson R. Decision support and disease management: A logic engineering approach. IEEE Transactions on Information Technology in Biomedicine 1998;2:217 – 228. 21. Fieschi M, Dufour J-C, Staccini P, Gouvernet J, Bouhaddou O. Medical decision support systems: Old dilemmas and new paradigms? Methods of Information in Medicine 2003;42:190 – 198. 22. Watt S. Clinical decision-making in the context of chronic illness. Health Expectations 2000;3:6 – 16. 23. Owens DK. Computer-based decision support: Wishing on a star? Effective Clinical Practice 2001;4:34 – 38. 24. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety. A systematic review. Archives of Internal Medicine 2003;163:1409 – 1416. 25. Poller L, Wright D, Rowlands M. Prospective comparative study of computer programs used for management of warfarin. Journal of Clinical Pathology 1993;46:299 – 303. 26. Evans RS, Classen DC, Pestotnik SL, Lundsgaarde HP, Burke JP. Improving empiric antibiotic selection using computer decision support. Archives of Internal Medicine 1994;154:878 – 884. 27. Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomised trial of ‘corollary orders’ to prevent errors of omission. Journal of the American Medical Informatics Association 1997;4:364 – 375. 28. Rossi RA, Every NR. A computerised intervention to decrease the use of calcium channel blockers in hypertension. Journal of General Internal Medicine 1997;12:672 – 678. 29. East TD, Heermann LK, Bradshaw RL, Lugo A, Sailors RM, Ershler L, Wallace J, Morris AH, McKinley B, Marquez A, Tonnesen A, Parmley L, Shoemaker W, Meade P, Thaut P, Hill T, Young M, Baughman J, Olterman M, Gooder V, Quinn B, Summer W, Valentine V, Carlson J, Bonnell B, deBoisblanc B, McClarity Z, Cachere J, Kovitz K, Gallacher E, Pinsky M, Angus D, Cohen M, Hudson L, Steinberg K. Efficacy of computerised decision support for mechanical ventilation: Results of a prospective multi-center randomized trial. In: Proceedings of the AMIA Symposium 1999. pp 251 – 255.
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
Computerised decision support on patient outcomes
239
30. Kuperman GJ, Teich JM, Tanasijevic MJ, Ma’Luf N, Rittenberg E, Jha A, Fiskio J, Winkelman J, Bates DW. Improving response to critical laboratory results with automation: Results of a randomised controlled trial. Journal of the American Medical Informatics Association 1999;6:512 – 522. 31. Chertow GM, Lee J, Kuperman GJ, Burdick E, Horsky J, Seger DL, Lee R, Mekala A, Song J, Komaroff AL, Bates DW. Guided medication dosing for inpatients with renal insufficiency. Journal of the American Medical Association 2001;286:2839 – 2844. 32. Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ. A computerised reminder system to increase the use of preventive care for hospitalised patients. New England Journal of Medicine 2001;345:965 – 970. 33. Mazzuca SA, Vinicor F, Einterz RM, Tierney WM, Norton JA, Kalasinski LA. Effects of the clinical environment on physicians’ response to postgraduate medical education. American Educational Research Journal 1990;27: 473 – 488. 34. Diabetes Integrated Care Evaluation Team (DICET). Integrated care for diabetes: clinical, psychological, and economic evaluation. British Journal of Medicine 1994;308:1208 – 1212. 35. Nilasena DS, Lincoln MJ. Computer-generated reminder system improves physician compliance with diabetes preventive care guidelines. In: Proceedings of the Annual Symposium on Computer Applications in Medical Care;1995. pp 640 – 645. 36. Lobach DF, Hammond WE. Computerized decision support based on clinical practice guidelines improves compliance with care standards. American Journal of Medicine 1997;102:89 – 98. 37. Hetlevik I, Holmen J, Kruger O, Kristensen P, Iversen H, Furuseth K. Implementing clinical guidelines in the treatment of diabetes mellitus in general practice. Evaluation of effort, process, and patient outcome related to implementation of a computer based decision support systems. International Journal of Technology Assessment in Health Care 2000;16:210 – 227. 38. Fitzmaurice DA, Hobbs FD, Murray ET, Holder RL, Allan TF, Rose PE. Oral anticoagulation management in primary care with the use of computerized decision support and near-patient testing: a randomized, controlled trial. Archives of Internal Medicine 2000;160:2343 – 2348. 39. Montgomery A, Fahey T, Peters T, MacIntosh C, Sharp D. Evaluation of computer based decision support system and risk chart for management of hypertension in primary care: randomised controlled trial. British Medical Journal 2000;320:686 – 690. 40. McCowan C, Neville RG, Ricketts IW, Warner FC, Hoskins G, Thomas GE. Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma. Medical Informatics and the Internet in Medicine 2001;26:191 – 201. 41. Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, Purves I. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. British Medical Journal 2002;325:941 – 944. 42. Rollman BL, Hanusa BH, Lowe HJ, Gilbert T, Kapoor WN, Schulberg HC. A randomized trial using computerized decision support to improve treatment of major depression in primary care. Journal of General Internal Medicine 2002;17:493 – 503. 43. Tamblyn R, Huang A, Perreault R, Jacques A, Roy D, Hanley J, McLeod P, Laprise R. The medical office of the 21st century (MOXXI): effectiveness of computerised decision-making support in reducing inappropriate prescribing in primary care. Canadian Medical Association Journal 2003;169:549 – 556. 44. Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou X-H, Eckert GJ, Smith FE, Nienaber N, McDonald CJ, Wolinsky FD. Effects of computerised guidelines for managing heart disease in primary care: A randomised controlled trial. Journal of General Internal Medicine 2003;18:967 – 976. 45. Weir CJ, Lees KR, MacWalter RS, Muir KW, Wallesch CW, McLelland EV, Hendry A, PRISM Study Group. Cluster-randomized, controlled trial of computer-based decision support for selecting long-term antithrombotic therapy after acute ischaemic stroke. Quarterly Journal of Medicine 2003;96:143 – 153. 46. Thomas HV, Lewis G, Watson M, Bell T, Lyons I, Lloyd K, Weich S, Sharp D. Computerised patient-specific guidelines for management of common mental disorders in primary care: A randomised controlled trial. British Journal of General Practice 2004;54:832 – 837. 47. Lester WT, Grant RW, Barnett GO, Chueh HC. Ramdomized controlled trial of an informatics-based intervention to increase statin prescription for secondary prevention of coronary disease. Journal of General Internal Medicine 2006;21:22 – 29. 48. Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou X-H, Eckert GJ, Smith FE, Nienaber N, McDonald CJ, Wolinsky FD. Can computer-generated evidence-based suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial. Health Services Research 2005;40(2):477 – 497.
Downloaded By: [University of Sydney] At: 00:18 27 September 2007
240
V. Sintchenko et al.
49. Ramnarayan P, Kapoor RR, Coren M, Nanduri V, Tomlinson AL, Taylor PM, Wyatt JC, Britto JF. Measuring the impact of diagnostic decision support on the quality of clinical decision making. Journal of the American Medical Informatics Association 2003;10:563 – 572. 50. Brender J, Ammenwerth E, Nykdnen P, Talmon J. Factors influencing success and failure of health informatics systems—A pilot Delphi study. Methods of Information in Medicine 2006;45(1):125 – 136. 51. Balas EA, Krishna S, Kretschmer RA, Cheek TR, Lobach DF, Boren SA. Computerized knowledge management in diabetes care. Medical Care 2004;42:610 – 621. 52. Jackson CL, Bolen S, Brancati FL, Batts-Turner ML, Gary TL. A systematic review of interactive computerassisted technology in diabetes care. Journal of General Internal Medicine 2006;21:105 – 110. 53. Coiera E. Maximising the uptake of evidence into clinical practice. Medical Journal of Australia 2001;174:467 – 470. 54. Benson T. Why general practitioners use computers and hospital doctors do not—Part 1: incentives. British Medical Journal 2002;325:1086 – 1089. 55. Wetter T. Lessons learnt from bringing knowledge-based decision support into routine use. Artificial Intelligence in Medicine 2002;24:195 – 203. 56. Sintchenko V, Coiera E. Which clinical decisions benefit from automation? A task complexity approach. International Journal of Medical Informatics 2003;70:309 – 316. 57. Coiera E. Four rules for the reinvention of health care. British Medical Journal 2004;328:1197 – 1199. 58. Dreiseitl S, Binder M. Do physicians value decision support? A look at the effect of decision support systems on physician opinion. Artificial Intelligence in Medicine 2005;33:25 – 30. 59. Thornett AM. Computer decision support systems in general practice. International Journal of Information Management 2001;21:39 – 47. 60. Trivedi MH, Kern JK, Marcee A, Grannemann B, Kleiber B, Bettinger T, Altshuler KZ, McClelland A. Development and implementation of computerized clinical guidelines: Barriers and solutions. Methods of Information in Medicine 2002;41:435 – 442. 61. Bankowitz RA, Lave JR, Mcneil MA. A method for assessing the impact of a computer-based decision support system on health outcomes. Methods of Information in Medicine 1992;31:3 – 11. 62. Feinstein AR. Clinimetrics. New Haven, CT: Yale University Press, 1987. 63. Shim JP, Warkentin M, Courtney JF, Power DJ, Sharda R, Calrsson C. Past, present, and future of decision support technology. Decision Support Systems 2002;33:111 – 126. 64. Hand DJ. Statistics and the theory of measurement. Journal of the Royal Statistical Society. Series A 1996;159:445 – 492. 65. Mant J. Process versus outcome indicators in the assessment of quality of health care. International Journal for Quality in Health Care 2001;13:475 – 480. 66. McCormick D, Fine MJ, Coley CM, Marrie TJ, Lave JR, Obrosky DS, Kapoor WN, Singer DE. Variation in length of hospital stay in patients with community-acquired pneumonia: are shorter stays associated with worse medical outcomes? American Journal of Medicine 1999;107:5 – 12. 67. Rothwell PM. External validity of randomised controlled trials: ‘To whom do the results of this trial apply?’ Lancet 2005;365:82 – 93. 68. Colombet I, Chatellier G, Jaulent M-C, Degoulet P. Decision aids for triage of patients with chest pain: A systematic review of field evaluation studies. In: Lorenzi NM, editor. Transforming Health Care Through Informatics/Proceedings of the Annual AMIA Symposium, 1999. pp 231 – 235.
