British Journal of Clinical Pharmacology
Br J Clin Pharmacol (2016) 82 1048–1057
1048
DRUG SAFETY Performance of trigger tools in identifying adverse drug events in emergency department patients: a validation study Correspondence Dr Corinne M. Hohl, Emergency Department, Vancouver General Hospital, 855 West 12 Avenue, Vancouver, British Columbia, V5Z 1M9, Canada. Tel.: 604 8751 ext.63467; Fax: 604 875 5506; E-mail:
[email protected] Received 7 February 2016; revised 19 May 2016; accepted 4 June 2016
Andrei Karpov1, Catherine Parcero2, Catherine P.Y. Mok1, Chandima Panditha2, Eugenia Yu3, Linda Dempster2 and Corinne M. Hohl1,4 1
Department of Emergency Medicine, University of British Columbia, Vancouver, British Columbia, V5Z 1M9, 2Quality & Patient Safety, Vancouver
Coastal Health, Vancouver, British Columbia, V5Z 1M9, 3Department of Statistics, University of British Columbia, Vancouver, British Columbia, V6T 1Z4, and 4Centre for Clinical Epidemiology and Evaluation, Vancouver Coastal Health Research Institute, Vancouver, British Columbia, V5Z 1M9, Canada
Keywords drug-related side effects and adverse reactions, emergency service, health care, hospital, quality indicators, trigger tools
AIMS Trigger tools are retrospective surveillance methods that can be used to identify adverse drug events (ADEs), unintended and harmful effects of medications, in medical records. Trigger tools are used in quality improvement, public health surveillance and research activities. The objective of the study was to evaluate the performance of trigger tools in identifying ADEs.
METHODS This study was a sub-study of a prospective cohort study which enrolled adults presenting to one tertiary care emergency department. Clinical pharmacists evaluated patients for ADEs at the point-of-care. Twelve months after the prospective study’s completion, the patients’ medical records were reviewed using eight different trigger tools. ADEs identified using each trigger tool were compared with events identified at the point-of-care. The primary outcome was the sensitivity of each trigger tool for ADEs.
RESULTS Among 1151 patients, 152 (13.2%, 95% confidence intervals (CI) 11.4, 15.3%) were diagnosed with one or more ADEs at the point-of-care. The sensitivity of the trigger tools for detecting ADEs ranged from 2.6% (95% CI 0.7, 6.6%) to 15.8% (95% CI 10.6, 22.8%). Their specificity varied from 99.3% (95% CI 98.6, 99.7) to 100% (95% CI 99.6, 100%).
CONCLUSION The trigger tools examined had poor sensitivity for identifying ADEs in emergency department patients, when applied manually and in retrospect. Reliance on these methods to detect ADEs for quality improvement, surveillance, and research activities is likely to underestimate their occurrence, and may lead to biased estimates.
DOI:10.1111/bcp.13032
© 2016 The British Pharmacological Society
Performance of trigger tools in identifying adverse drug events
WHAT IS ALREADY KNOWN ABOUT THIS SUBJECT • Trigger methods are widely used to identify adverse drug events and adverse drug reactions. • Trigger methods have not been compared with prospective identification of adverse drug events to understand their completeness and accuracy
WHAT THIS STUDY ADDS • The current available trigger tools have poor sensitivity for identifying adverse drug events in emergency department patients. • More robust methods for the detection and monitoring of adverse drug events are needed.
Introduction Adverse drug events (ADEs), unintended and harmful events related to medication use, are a common source of patient injury within healthcare [1, 2]. They are a leading cause of ambulatory and emergency department visits, unplanned hospital admissions and deaths [3–6]. Patients who present to emergency departments with ADEs spend more days in hospital and incur nearly double the healthcare costs over 6 months compared with patients presenting without medication-related morbidity [7]. The majority of ADEs are, in retrospect, deemed preventable [5, 8]. However, identifying and implementing practice changes that result in sustained and measurable reductions of ADEs has proven challenging: Efforts directed at reducing medication errors—failures in the process of medication delivery, have not optimized the safety of healthcare, in part because many errors are intercepted before reaching the patient [1, 9, 10]. In contrast, medications administered without error may result in harm due to toxic reactions or dosing problems [11]. It is possible that greater improvements in patient safety may be achieved by changing the focus of preventative efforts from error to harm reduction [1]. Until recently, a major barrier to developing evidencebased harm reduction strategies was the inability of healthcare institutions to identify and monitor ADEs over time in an efficient and consistent manner, so that data could inform the development and evaluation of quality improvement strategies [1]. Conventional approaches to identifying ADEs included voluntary reporting, chart reviews and mining administrative data using patient safety indicators and discharge diagnoses. These methods are resource-intensive, provide inconsistent results over time, or significantly under report patient safety events, prompting continued investigation into more practical and efficient methods to identify adverse outcomes [12–16]. An alternate method for identifying ADEs was described in 1974 and has since been adapted to various healthcare settings [17]. Trigger tools use a two step chart review methodology. In a first step, trained reviewers screen a random selection of patient records and flag those containing a trigger or clue, that may consist of an individual word, phrase, order or laboratory value [1, 17, 18]. In a second step, a trained nurse or physician reviews the flagged records to determine whether an adverse event occurred. Today, trigger methods are in widespread use in quality improvement and increasingly in pharmacosurveillance and research activities [17, 19]. For example, the National Electronic Injury Surveillance System-
Cooperative Adverse Drug Event Surveillance Project (NEISSCADES) uses trigger methods to describe and monitor outpatient ADEs treated in USA emergency departments [19, 20]. Despite their widespread application, trigger methods have not been compared to prospective identification of ADEs to understand their completeness and accuracy. Therefore, our aim was to evaluate the sensitivity and specificity of published trigger tools in identifying adverse drugs events in comparison with events identified in a prospective sample of adult emergency department patients.
Methods Study setting and design This study was an a priori planned sub-study of a prospective observational study to derive clinical decision rules to identify emergency department patients at high-risk of ADEs [21]. This study was conducted in the one Canadian tertiary care hospital, Vancouver General Hospital, with an annual census of 84 000 patients. The University of British Columbia Clinical Research Ethics Board (H10-01632) reviewed and approved the study protocol and waived the need for informed consent.
Enrolment and data collection The patient enrolment and data collection procedures for the parent study have been described in detail elsewhere [21]. Briefly, patients presenting to the emergency department between July 1 2008 and January 24 2009 were eligible for enrolment. Within each data collection shift, clinical pharmacists applied a systematic patient selection algorithm to ensure a representative sample of patients (see Supplemental Figure, Appendix S1) [5]. All patients 19 years of age or older who reported using at least one prescription or over the counter medication in the 2 weeks prior to presentation and spoke English or had a translator available were deemed eligible. Patients were excluded if they were transferred directly to an admitting service, presented for a scheduled revisit (e.g. intravenous antibiotics), left against medical advice or before the pharmacist assessment was complete, exhibited violent behaviour or had previously been enrolled. After prospective data collection was complete, we excluded patients if their medical record could not be located or if data on inclusion and exclusion criteria were missing. Three clinical pharmacists collected demographic and clinical information from patients at the point-of-care, and Br J Clin Pharmacol (2016) 82 1048–1057
1049
A. Karpov et al.
reviewed the medical record in the emergency department. They verified medication histories using PharmaNet, a province wide prescription filling database. All admitted patients were followed until hospital discharge and consenting patients were contacted after discharge by telephone when necessary to determine whether the patient met the primary or secondary outcome. Pharmacists evaluated whether the patient’s visit was due to an ADE arising prior to the patient’s presentation to the Emergency Department, using three standardized causality algorithms [22–24]. Inter-rater reliability of this assessment algorithm was evaluated to ensure reliability and was previously reported (kappa 0.75, 95% CI 0.52, 0.98) [21]. The pharmacists then interviewed treating physicians using a standardized questionnaire to determine whether the physician believed the patient had suffered an ADE. An independent committee adjudicated all discordant or uncertain cases to establish the final diagnosis [21].
Assessment of trigger methods Research assistants (AK and CM), who had not been involved with the primary study, manually reviewed the medical records of all enrolled patients 6–12 months after the date of their emergency department visit and after all admitted patients were discharged. This included a review of both the electronic records as well as the paper chart of included patients. We applied the two step chart review methodology described by the Institute for Healthcare Improvement [17, 19]. First, we piloted a data collection form containing the individual triggers of eight published trigger tools (Appendix S2) [1, 17, 18, 25–28], omitting triggers for non-medicationrelated adverse events, triggers not applicable to our patient population and the triggers for medications that had not been prescribed in our cohort of patients. The final form contained 63 individual triggers and the NEISS-CADES algorithm which contains the triggers allergic reaction, adverse effect, side effect, secondary to, ingestion, toxicity, medication error, angioedema, anaphylaxis, rash, bleeding and hypoglycaemia [19]. We trained two research assistants in the use of the data collection form (AK and CM) and randomly selected 21 charts for review to measure the inter-rater agreement between their assessments. The research assistants reviewed the medical records of all patients to identify individual triggers and determine whether the NEISS-CADES criteria were met. Records for which one or more triggers were flagged as having been met proceeded to the second stage review. In the second review stage, a trained nurse specializing in patient safety and quality (CP) who applies trigger tools in her work reviewed the flagged charts for the presence of an ADE. The first and second stage reviewers were blinded to the ADE determination from the primary study. A trigger tool or the NEISS-CADES algorithm were considered to have flagged an ADE,,if any of its component triggers were present during the first review step and the second review step identified an ADE (see Appendix S4 for a study flow chart).
Outcome measures The ADE outcomes identified in the primary study were considered the criterion standard for the sub-study. ADEs were defined as ‘utoward and unintended symptoms, signs or abnormal laboratory values arising from the appropriate or 1050
Br J Clin Pharmacol (2016) 82 1048–1057
inappropriate use of prescription or over the counter medications’ [13, 21]. Pharmacists classified ADEs into seven categories: (i) Adverse drug reactions, defined as ‘noxious and/or unintended responses to medication which occurred despite appropriate drug dosage for prophylaxis, diagnosis or therapy of the indicating medical condition’ [29], events caused by (ii) non-adherence, (iii) subtherapeutic dosing, (iv) supratherapeutic dosing, (v) receiving the wrong drug, (vi) suffering from an untreated indication or (vii) using a drug without a treatment indication [11, 30]. The pharmacists individually assessed causality at the point-of-care using previously adapted standardized algorithms, ruling out ADEs for any events in which physicians identified alternate diagnoses [22–24], and assessed preventability using criteria developed by Hallas et al. [31]. We rated the severity of the events as severe if the event caused death or required hospital admission, moderate if the event required a change in medical management and mild if the event required no change in management [5]. ADEs arising from the administration of medications in the Emergency Department or in hospital were not included in the analysis.
Statistical analysis We used descriptive statistics to summarize the baseline characteristics of the patient population. We reported the inter-rater reliability of collecting data on individual triggers by using the prevalence adjusted and biased adjusted kappa (PABAK) score [32]. We estimated the sensitivity of the trigger tools by dividing the number of records with one or more ADEs identified using the trigger tool by the total number of records with ADEs identified during the prospective study, multiplied by 100 [33]. We examined the specificity of the trigger tools by dividing the number of records that screened negative using the trigger tool without ADEs by the total number of charts without ADEs, multiplied by 100. We decided a priori to conduct a sensitivity analysis and calculate the sensitivities and specificities of the tools for moderate and severe adverse drug reactions. We reported 95% CIs of the sensitivity and specificity.
Results Adverse drug events identified at the point-of-care Among 1566 patients who were approached for enrolment, 1160 were included in the prospective study (Figure 1) [21]. The charts of eight patients could not be located at the time of data collection for the trigger tool evaluation and two additional charts were unavailable at the time of the NEISSCADES evaluation. One additional record had insufficient data to determine the inclusion and exclusion criteria and was excluded. The baseline characteristics of the study population are listed in Table 1. Among 1151 patients, 152 (13.2%, 95% CI 11.4, 15.3%) were diagnosed with 164 ADEs at the point-ofcare (Table 2). Clinical pharmacists classified 34.8% (57/ 164) of these events as adverse drug reactions, 66.5% (109/ 164) of them as being directly related to the patient’s chief
Performance of trigger tools in identifying adverse drug events
98.6, 99.7%) to 100% (95% CI 99.6, 100%). Their positive predictive values were consistent between tools and ranged between 57.1% (95% CI 18.4, 90.1%) and 100% (95% CI 63.1, 100%). Their negative predictive values varied between 87.1% (95% CI 84.9, 88.9%) and 88.2% (95% CI 86.1, 90.0%).
Evaluation of the NEISS-CADES algorithm The kappa scores for the inter-rater agreement between the first stage reviewers with regard to whether a NEISS-CADES trigger was present or absent was 1 (95% CI 1, 1). The NEISS-CADES algorithm had a sensitivity of 15.8% (95% CI 10.6, 22.8%) and a specificity of 99.5% (95% CI 98.8, 99.8%) for detecting records with at least one ADE and a sensitivity of 38.8% (95% CI 25.5, 53.8%) and specificity of 99.1% (95% CI 98.3, 99.6%) for moderate or severe adverse drug reactions (Table 4). The NEISS-CADES algorithm missed 137 (83.5%) of the 164 prospectively identified ADEs and 100 (87.7%) of preventable events. The algorithms’ positive predictive value was 82.8% (95% CI 64.2, 94.2%) and its negative predictive value 88.6% (95% CI 86.6, 90.4%)
Discussion Figure 1 Flow diagram of included patients
complaint and 69.5% (114/164) as preventable. Most events (81.1%, 133/164) were rated as moderate in severity and warfarin, paracetamol (acetaminophen) with codeine, aspirin, phenytoin, olanzapine and hydrochlorothiazide were the most commonly implicated medications.
Evaluation of trigger tools The kappa scores for the inter-rater agreement between the first stage reviewers with regard to the presence or absence of individual triggers ranged from 0.81 (95% CI 0.55, 1) to 1 (95% CI 1, 1). Appendix S3 (see supplemental table, Appendix S3) describes the number of records that were flagged as positive based on the individual triggers. The most commonly flagged triggers were >6 h in the emergency department, unplanned hospitalization or transfer to a higher level of care, rising serum creatinine, rising blood urea nitrogen level or creatinine greater than twice baseline and the use of an antiemetic. Table 3 presents the diagnostic test characteristics of the trigger tools we evaluated. The sensitivities of the trigger tools for records containing one or more ADEs ranged from 2.6% (95% CI 0.7, 6.6%) to 12.5% (95% CI 7.9, 19.1%). For records with at least one moderate or severe adverse drug reaction their sensitivities ranged from 4.1% (95% CI 0.5, 14.0%) to 22.5% (95% CI 12.2, 37.0%). Using the flags of the individual trigger tools as a basis for conducting the second stage review, between 140 (87.2%) and 157 (95.7%) of the 164 ADEs were missed. These methods also missed between 101 (88.6%) and 113 (99.1%) of the 114 ADEs classified as preventable. The specificities of the tools ranged from 99.3% (95% CI
Our objective was to evaluate the classification performance of eight published trigger tools for ADE identification in a sample of adult emergency department patients. To our knowledge, our study is the first to validate trigger tools by comparing adverse events identified by the tools to events identified at the point-of-care by clinical care providers. Our findings indicate that the sensitivities of the trigger tools and the NEISS-CADES algorithm were uniformly low, ranging from 2.6% to 15.8% compared with the prospective standard. All trigger tools missed the majority of ADEs deemed preventable. In contrast, their specificities for ADEs were high, indicating few false positive cases. Over the past 15 years, trigger tools have been widely adopted by healthcare institutions internationally to identify and monitor adverse events related to medical care [17]. Their implementation followed recognition of their feasibility and ease of use and was driven by a pressing need to find and apply new methods to capture adverse event data for quality improvement [17, 34]. As ‘big data’ from comprehensive electronic health records became available for data mining, trigger tools were also applied as a means of identifying adverse outcomes within large databases for surveillance and research [35]. However, the rapid and widespread uptake of trigger methods preceded their rigorous validation, which is intended to ensure that the intended outcomes are identified consistently, completely and accurately [36]. The practice of generating and analyzing trigger-derived data in advance of robust validation is problematic as it limits our ability to understand their inherent limitations. The assessment of any trigger tool’s sensitivity, its ‘miss’ rate, requires a comparison of outcomes identified using the tool with events identified by an independent robust criterion standard for all patients within a study sample [36]. Previous attempts at validating trigger tools have used retrospective methods, voluntary reporting and administrative data to identify adverse event outcomes [16, 33, 35, 37–41]. This Br J Clin Pharmacol (2016) 82 1048–1057
1051
A. Karpov et al.
Table 1 Characteristics of included patients Characteristics
All patients (n = 1151)
Patients without adverse drug events (n = 999)
Patients with ≥1 adverse drug event (n = 152)
Age (years) mean (SD)
51.6 (20.9)
50.9 (20.6)
56.1 (21.7)
Female
610 (53.0%)
538 (53.9%)
72 (47.4%)
Male
541 (47.0%)
461 (46.1%)
80 (52.6%)
Home
1048 (91.1%)
917 (91.9%)
131 (86.2%)
Homeless or shelter
29 (2.5%)
19 (1.9%)
10 (6.6%)
Nursing home
49 (4.3%)
40 (4.0%)
9 (5.9%)
Other
24 (2.1%)
22 (2.2%)
2 (1.3%)
Speak English
1112 (96.6%)
968 (96.9%)
144 (94.7%)
Translator available
39 (3.4%)
31 (3.1%)
8 (5.3%)
Acute care
509 (44.3%)
406 (40.8%)
103 (67.8%)
Minor care
639 (55.7%)
590 (59.2%)
49 (32.2%)
Gender, n (%)
Arrived from, n (%)
English language, n (%)
Emergency department treatment location, n (%)
Canadian triage acuity score, n (%) 1
5 (0.43%)
3 (0.30%)
2 (1.3%)
2
174 (15.1%)
150 (15.0%)
24 (15.8%)
3
507 (44.0%)
423 (42.3%)
84 (55.3%)
4
425 (36.9%)
387 (38.7%)
38 (25.0%)
5
40 (3.47%)
36 (3.6%)
4 (2.6%)
Abdominal pain
103 (8.6%)
96 (9.6%)
7 (4.6%)
Chest pain
77 (6.7%)
72 (7.2%)
5 (3.3%)
Shortness of breath
68 (5.9%)
55 (5.5%)
13 (8.6%)
Lower extremity pain
57 (5.0%)
54 (5.4%)
3 (2.0%)
Back pain
51 (4.4%)
47 (4.7%)
4 (2.6%)
Number of prescription medications, median (IQR)
2 (1,5)
2 (1,5)
4 (2,7)
CAM use, n (%)
137 (12.0%)
116 (11.7%)
21 (13.8%)
Over the counter medication use, n (%)
853 (74.6%)
758 (76.5%)
95 (62.5%)
Most common chief complaints, n (%)
Number of prescribing physicians, median (IQR)
1 (1, 2)
1 (1, 2)
2 (1, 2)
Number of comorbid conditions, median(IQR)
2 (1, 3)
1 (0, 3)
2 (1, 4)
Followed by a general practitioner, n (%)
993 (87.0%)
859 (86.9%)
134 (88.2%)
Home
936 (81.4%)
829 (83.1%)
107 (70.4%)
Admitted
210 (18.3%)
167 (16.7%)
43 (28.3%)
Transferred
3 (0.26%)
1 (0.1%)
2 (1.3%)
Died in ED, n (%)
1 (0.09%)
1 (0.1%)
0 (0%)
Disposition from emergency department, n (%)
CAM, complimentary or alternative medications; ED emergency department
1052
Br J Clin Pharmacol (2016) 82 1048–1057
Performance of trigger tools in identifying adverse drug events
Table 2 Characteristics of 164 adverse drug events identified at the point-ofcare in 152 patients
Characteristics
Adverse drug events (n = 164)
Severity, n (%) Severe
21 (12.8%)
Moderate
133 (81.1%)
Mild
10 (6.1%)
Preventability, n (%) Preventable
114 (69.5%)
Non-preventable
50 (30.5%)
Relationship to chief complaint, n (%) Chief complaint-related
109 (66.5%)
Incidentally found
55 (33.5%)
Classification, n (%) Adverse drug reactions
57 (34.8%)
Non-compliance or drug withdrawal
36 (22.0%)
Untreated indication
26 (15.9%)
Subtherapeutic dose
21 (12.8%)
Wrong drug
10 (6.1%)
Supratherapeutic dose
9 (5.5%)
Drug without indication
4 (2.4%)
Drug interactions
1 (0.6%)
Most common culprit medications, n (%) Warfarin
14 (8.5%)
Paracetamol with codeine
11 (6.7%)
Aspirin
9 (5.5%)
Phenytoin
7 (4.3%)
Olanzapine
6 (3.7%)
Hydrochlorothiazide
6 (3.7%)
Cephalexin
5 (3.0%)
Hydromorphone
5 (3.0%)
Glyburide
4 (2.4%)
Morphine
4 (2.4%)
is problematic, as adverse event outcomes are poorly identifiable within medical records due to incomplete documentation, underreported within existing adverse event reporting platforms and poorly identifiable within administrative data [12–16, 42]. Therefore, none of these methods is a robust standard for comparison and, as a result, led to overestimations of sensitivity [36]. In contrast to previous validation
attempts, our study is the first to compare the performance of trigger tools to independent, prospectively established outcomes. Five previous studies reported the sensitivity of the same trigger tools we assessed and reported sensitivities ranging from 33–94.9 % [16, 33, 39, 41, 43]. Two focused on adverse event identification in adults [16, 33] and paediatrics [41], and two on ADE identification in adults [39, 43]. Previous studies are likely to have overestimated the sensitivity of the trigger tools due to incomplete outcome determinations and, in some cases, outcome determinations not being independent of the trigger tool application. This occurred when the same trigger methods were applied to pre-screen charts or find the outcomes of interest [16, 33, 39]. All previous validation studies using retrospective methods have missed all undocumented outcomes [16, 33, 39, 41, 43]. In one study, the same research pharmacist completed the identification of ADE outcomes prior to applying the trigger tool and, therefore, would have been biased towards finding a higher number of events [43]. Another used the outcomes identified by the trigger method being evaluated, combined with outcomes identified by voluntary reporting and application of indicators in administrative data [16]. For these reasons, none of the previous studies reporting sensitivity meet the methodological standards required to provide unbiased estimates of sensitivity. Most previous attempts at validating trigger tools have focused on their positive predictive value, the proportion of flagged or trigger positive records containing the outcome of interest [33, 35, 37, 39, 41, 43–46]. This metric is useful in providing an estimate of the yield within the flagged records and enables an estimate of the excess workload generated by false positive flags [35]. Since trigger tools are used as a screening method and are meant to be applied to large data sets, trigger tools with high positive predictive values are key in minimizing the false positive rate, as long as the sensitivity of these tools is maintained to a reasonable degree. In our study, the positive predictive value of the tools varied between 57.1–100%, consistent with ranges reported in previous studies. A few studies have reported lower positive predictive values when evaluating a subset of laboratory based triggers, signalling adverse events in only 15% of cases [45], or adaptations of existing tools, signalling events in 4% and 17% of cases [43, 46, 47]. The latter studies call into question the signal-to-noise ratio and any efficiency gained by employing trigger tool methodology to prescreen records. In our study, we found high inter-rater reliability of individual triggers compared with the level of agreement reported by others [33, 40, 41, 44, 46, 48]. This may be due to multiple factors. We trained research assistants extensively, attempted to discuss and minimize the degree of subjective interpretation required to apply the triggers in advance of piloting and asked research assistants to review the emergency department record of only one hospital, minimizing variation and shortening the length of time required to complete the review. As a result, our reviewers were able to evaluate the complete emergency department record within the set time limit suggested by the Institute for Healthcare Improvement [48]. Finally, the sample from which we report inter-rater agreement is small, and thus our estimates are more uncertain compared with other studies. Br J Clin Pharmacol (2016) 82 1048–1057
1053
A. Karpov et al.
Table 3 Diagnostic performance characteristics of the trigger methods for adverse drug events Patients with >1 adverse drug event Trigger tool
Target population
Sensitivity % (95% CI)
Specificity % (95% CI)
PPV % (95% CI)
NPV % (95% CI)
GlobalGriffin & Resar [17]
Hospitalized
12.5% (7.9, 19.1)
99.3% (98.6, 99.7)
73.1% (52.1, 88.4)
88.2% (86.1, 90.0)
GlobalResar et al. [1]
Hospitalized
10.5% (6.3, 16.8)
99.5% (98.8, 99.8)
76.2% (52.8, 91.8)
88.0% (85.9, 89.8)
Adverse drug event Singh et al. [26]
Ambulatory
11.2% (6.8, 17.6)
99.7% (99.1, 99.9)
85.0% (62.1, 96.8)
88.1% (86.0, 90.0)
Adverse drug event Rozich [18]
Hospitalized
12.5% (7.9, 19.1)
99.6% (99.0, 99.9)
82.6% (61.2, 95.1)
88.2% (86.1, 90.0)
Adverse drug event Hug [28]
Hospitalized
12.5% (7.9-19.1)
99.6% (99.0, 99.9)
82.6% (61.2, 95.1)
88.2% (86.1, 90.0)
Adverse drug reaction-Cantor et al. [25]
Ambulatory
5.3% (2.3-10.1)
100% (99.6, 100)
100% (63.1, 100)
87.4% (85.3, 89.2)
Electronic-Wolff [27]
ED
2.6% (0.7, 6.6)
99.7% (99.1-99.9)
57.1% (18.4, 90.1)
87.1% (84.9, 88.9)
NEISS-CADES [3]
ED
15.8 (10.6-22.8)
99.5% (98.8, 99.8)
82.8% (64.2, 94.2)
88.6% (86.6, 90.4)
ADE adverse drug event; ADR adverse drug reaction; ED Emergency Department; NEISS-CADES National Electronic Injury Surveillance System-Cooperative Adverse Drug Events Surveillance System; PPV, positive predictive value
Table 4 Number and proportion of missed adverse drug events, by event characteristics and trigger method used
Trigger method used for screening
Adverse drug Moderate adverse Severe adverse Preventable adverse Chief-complaint related events (n = 164) drug events (n = 133) drug events (n = 21) drug events (n = 114) adverse drug events (n = 109)
Griffin & Resar [17], n (%)
140 (87.4%)
115 (85.8%)
15 (71.4%)
101 (88.6%)
93 (85.3%)
Singh et al. [26], n (%)
143 (87.2%)
118 (88.7%)
15 (71.4%)
103 (90.4%)
94 (86.2%)
Rozich et al. [18], n (%)
140 (87.4%)
115 (85.8%)
15 (71.4%)
101 (88.6%)
93 (85.3%)
Hug et al. [28], n (%)
140 (87.4%)
115 (85.8%)
15 (71.4%)
101 (88.6%)
93 (85.3%)
Cantor et al. [25], n (%)
154 (93.9%)
123 (92.5%)
21 (100%)
108 (94.7%)
101 (92.7%)
Resar et al. [1], n (%) 144 (87.8%)
119 (89.5%)
15 (71.4%)
104 (91.2%)
95 (87.2%)
Wolff & Bourke [27], 157 (95.7%) n (%)
128 (96.2%)
19 (90.5%)
113 (99.1%)
105 (96.3%)
NEISS-CADES [19], n (%)
113 (85.0%)
15 (71.4%)
100 (87.7%)
90 (82.6%)
137 (83.5%)
NEISS-CADES National Electronic Injury Surveillance System- Cooperative Adverse Drug Event Surveillance
In our team’s work in deriving clinical decision rules to predict patients at high risk for ADEs, drug and disease specific variables were not useful, as was seen in this study [21]. This was likely because ADEs tend to be heterogeneous in nature. The trigger tools developed so far contain a large number of disease and drug-specific manifestations, laboratory values or antidotes, which in our analysis were not discriminatory. In our work on predicting ADEs, our team found that drug and disease specific variables predicted too small a subset of events to be useful, e.g. a trigger using an INR cutoff was only useful for events related to warfarin, which constituted an important but small minority of events. However, more generic 1054
Br J Clin Pharmacol (2016) 82 1048–1057
variables such as ‘medication change within 28 days’ could predict multiple types of ADEs, because harm generally occurred shortly after a drug had been initiated. We believe that more sensitive trigger tools might be developed by using clinical decision rule methodology in which clinical judgment is initially used to define lists of candidate predictor variables. Subsequently, a prospective observational study is conducted in which data on candidate predictor variables and the study outcome are collected. Then, statistical associations between the candidate predictor variables and outcome are carefully explored in a systematic way to parse out potentially useful variables and combine them to a decision algorithm [36].
Performance of trigger tools in identifying adverse drug events To our knowledge, our study represents the first external validation of the NEISS-CADES algorithm, which is presently in use to monitor and describe the public health burden of outpatient ADEs treated in USA emergency departments [49]. We estimated its sensitivity at 15.8%, indicating that 84.2% of ADEs may be missed in this important public health surveillance system. When a narrower case definition was applied, focusing on moderate and severe adverse drug reactions only, its sensitivity was increased to 38.8%. However, NEISS-CADES missed the majority of preventable ADEs, indicating that opportunities for prevention and system-wide change may be missed if we rely solely on this case finding method [27]. Our findings have important implications for quality improvement initiatives, public health surveillance and research efforts that use trigger-derived data as a means of developing and evaluating strategies to improve clinical care. In 2010, Landrigan et al. published the results of a longitudinal study evaluating whether investments in quality improvement strategies over 5 years led to a measurable improvement in adverse event rates in a sample of hospitals in North Carolina [50]. The study found little evidence of improvement over time and concluded that harm remained common. The results of our study may offer additional insight into these findings. In our study, the majority of preventable ADEs were not identifiable using the trigger methods we evaluated. Thus, it is possible that the poor sensitivity of the global trigger tool for preventable events may have limited the authors’ ability to detect improvement. Alternatively, if the healthcare institutions used trigger-derived data to prioritize and develop quality improvement strategies, it is plausible that few preventable adverse outcomes were in fact targeted, and that the programs did in fact have less impact on safety than anticipated.
Limitations Our study is not without limitations. Our study was conducted in one academic centre and,therefore, our results may not be generalizable to other types of institutions. We evaluated all tools against ADE cases that presented to emergency departments, even though some tools were specifically developed for other healthcare settings. It is possible that the sensitivity of the tools may be greater for other types of hospitals and care settings. However, two of the tools we evaluated were specifically designed for emergency departments and one had the highest while the other the lowest sensitivity for ADEs. Among the tools we evaluated, three were designed for adverse events while five had been developed to identify adverse events to medications. This was likely reflected in the lower positive predictive value of those tools not specifically designed for ADEs. Differences between the definitions of ADE used in developing the trigger tools and that used in our study may have contributed to the low sensitivity we found. For this reason we completed a sensitivity analysis, in which we used the narrowest definition for ADEs possible, adverse drug reactions, and excluded all mild events. Even in this analysis the sensitivity of the trigger tools remained low. In conclusion our results suggest that eight commonly used trigger tools, including the NEISS-CADES algorithm currently used for public health surveillance of outpatient ADEs in the US, suffer from poor sensitivity. The majority of missed
ADEs were preventable, suggesting the need for continued development of more robust methods to detecting and monitor ADEs. Our results highlight the importance of validating retrospective ADE case finding methods against a robust prospective standard, so that refinement may be attempted prior to their widespread implementation.
Competing Interests All authors completed the Unified Competing Interest form at http://www.icmje.org/coi_disclosure.pdf and declare no support from any organization for the submitted work, no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years and no other relationships or activities that could appear to have influenced the submitted work.
References 1 Resar RK, Rozich JD, Classen D. Methodology and rationale for the measurement of harm with trigger tools. Qual Saf Health Care 2003; 12: ii39–ii45. 2 Institute of Medicine. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press, 2000. 3 Budnitz D. National Injury Surveillance System - Cooperative Adverse Drug Event Surveillance Project (NEISS-CADES). 2011. Available at http://www.c-path.org/pdf/BudnitzPSSW.pdf (last accessed 20 December 2011). 4 Baker GR, Norton P. Making patients safer! Reducing error in Canadian healthcare. Healthc Pap 2001; 2: 10–31. 5 Zed PJ, Abu-Laban RB, Balen RM, Loewen PS, Hohl CM, Brubacher JR, et al. Incidence, severity and preventability of medicationrelated visits to the emergency department: a prospective study. Can Med Assoc J 2008; 178: 1563–1569. 6 Lazarou J, Pomeranz B, Corey P. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA 1998; 279: 1200–1205. 7 Hohl CM, Nosyk B, Zed P, Kuramoto L, Sobolev B, Brubacher J, et al. Outcomes of emergency department patients presenting with adverse drug events. Ann Emerg Med 2011; 58: 270–279. 8 McDonnell PJ, Jacobs MR. Hospital admissions resulting from preventable adverse drug reactions. Ann Pharmacother 2002; 36: 1331–1336. 9 Mueller SK, Sponsler KC, Kripalani S, Schnipper JL. Hospital-based medication reconciliation practices. Arch Intern Med 2012; 172: 1057–1069. 10 Kwan JL, Lo L, Sampson M, Shojania KG. Medication reconciliation during transitions of care as a patient safety strategy. Ann Intern Med 2013; 158: 397–403. 11 Nebeker J, Barach P, Samore M. Clarifying adverse drug events: a clinician’s guide to terminology, documentation, and reporting. Ann Intern Med 2004; 140: 795–801. 12 Wiktorowicz M, Lexchin J, Moscou K, Silversides A, Eggertson L. Keeping an eye onprescription drugs, keeping Canadians safe: active monitoring systems for drug safety and effectiveness in Canada and internationally. 2010. Available at http://
Br J Clin Pharmacol (2016) 82 1048–1057
1055
A. Karpov et al.
publications.gc.ca/collections/collection_2011/ccs-hcc/H17421-2010-eng.pdf (last accessed 13 October 2013).
30 Hepler C, Strand L. Opportunities and responsibilities in pharmaceutical care. Am J Hosp Pharm 1990; 47: 533–543.
13 Hohl CM, Kuramoto L, Yu E, Rogula B, Stausberg J, Sobolev B. Evaluating adverse drug event reporting in administrative data from emergency departments: a validation study. BMC Health Serv Res 2013; 13: 473–484.
31 Hallas J, Harvald B, Gram LF, Grodum E, Brosen K, Haghfelt T, et al. Drug related hospital admissions: the role of definitions and intensity of data collection, and the possibility of prevention. J Intern Med 1990; 228: 83–90.
14 Hohl CM, Robitaille C, Lord V, Dankoff J, Colacone A, Pham L, et al. Emergency physician recognition of adverse drug-related events in elder patients presenting to an emergency department. Acad Emerg Med 2005; 12: 197–205.
32 Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 327: 307–310.
15 Hohl CM, Zed PJ, Brubacher JR, Abu-Laban RB, Loewen PS, Purssell R. Do emergency physicians attribute drug-related emergency department visits to medication-related problems? Ann Emerg Med 2010; 55: 493–502. 16 Classen DC, Resar R, Griffin F, Federico F, Frankel T, Kimmel N, et al. ‘Global trigger tool’ shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff 2011; 30: 581–589. 17 Griffin F, Resar R. IHI Global Trigger Tool for Measuring Adverse Events. IHI Innovation Series white paper. Cambridge, MA: Institute for Healthcare Improvement, 2009. 18 Rozich J, Haraden C, Resar R. Adverse drug event trigger tool: a practical methodology for measuring medication related harm. Qual Saf Health Care 2003; 12: 194–200. 19 Budnitz DS, Pollock DA, Weidenbach KN, Mendelsohn AB, Schroeder TJ, Annest JL. National surveillance of emergency department visits for outpatient adverse drug events. JAMA 2006; 296: 1858–1866. 20 Schaefer MK, Shehab N, Cohen AL, Budnitz DS. Adverse events attributable to cough and cold medications in mhild. Pediatrics 2008; 121: 1–5. 21 Hohl CM, Brubacher J, Hunte G, Wiens MO, Abu-Laban RB, Singer J, et al. Clinical decision rules to improve the detection of adverse drug events in emergency department patients. Acad Emerg Med 2012; 19: 640–649. 22 Forster A, Asmis T, Clark H, Al Saied G, Code CC, Caughey SC, et al. Ottawa Hospital Patient Safety Study: incidence and timing of adverse events in patients admitted to a Canadian teaching hospital. Can Med Assoc J 2004; 170: 1235–1240. 23 Naranjo CA, Busto U, Sellers EM, Sandor P, Ruiz I, Roberts EA, et al. A method for estimating the probability of adverse drug reactions. Clin Pharmacol Ther 1981; 30: 239–245. 24 WHO. Letter MIO/372/2(A). 1991. 25 Cantor MN, Feldman HJ, Triola MM. Using trigger phrases to detect adverse drug reactions in ambulatory care notes. Qual Saf Health Care 2007; 16: 132–134. 26 Singh R, McLean-Plunckett EA, Kee R. Experience with a trigger tool for identifying adverse drug events among older adults in ambulatory primary care. Qual Saf Health Care 2009; 18: 199–204. 27 Wolff AM, Bourke J. Detecting and reducing adverse events in an Australian rural base hospital emergency department using medical record screening and review. Emerg Med J 2002; 19: 35–40. 28 Hug LB, Witkowski DJ, Sox CM, Keohane CA, Seger DL, Yoon C, et al. Adverse drug event rates insix community hospitals and the potential impact of computerized physician order entry for prevention. J Gen Intern Med 2009; 25: 31–38. 29 World Health Organization. International drug monitoring: the role of the hospital. Report of a World Health Organization Meeting. World Health Organ Tech Rep Ser 1966; 425: 5–24.
1056
Br J Clin Pharmacol (2016) 82 1048–1057
33 Sharek PJ, Parry G, Goldmann D, Bones K, Hackbarth A, Resar R, et al. Performance characteristics of a methodology to quantify adverse events over time in hospitalized patients. Health Serv Res 2011; 46: 654–678. 34 Institute for Healthcare Improvement. Trigger tool for measuring adverse drug events. Institute for Healthcare Improvement and Premier, 2004 35 Murphy DR, Laxmisan A, Reis BA, Thomas EJ, Esquivel A, Forjuoh SN, et al. Electronic health record-based triggers to detect potential delays in cancer diagnosis. BMJ Qual Saf 2014; 23: 8–16. 36 Stiell I, Wells G. Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med 1999; 33: 437–447. 37 Kalenderian E, Walji MF, Tavares A, Ramoni RB. An adverse event trigger tool in dentistry: A new methodology for measuring harm in the dental office. J Am Dent Assoc 2013; 144: 808–814. 38 Kirkendall ES, Kloppenborg E, Papp J, White D, Frese C, Hacker D, et al. Measuring adverse events and levels of harm in pediatric inpatients with the global trigger tool. Pediatrics 2012; 130: e1206. 39 Nelson T, Budnitz D, Weidenbach K, Kegler SR, Pollock DA, Mendelsohn AB. Assessing the National Electronic Injury Surveillance System -- Cooperative Adverse Drug Event Surveillance Project - Six sites, United States, January 1–310 June 15, 2004. MMWR Weekly 2005; 54: 380–383. 40 Naessens JM, O’Byrne TJ, Johnson MG, Vansuch MB, McGlone CM, Huddleston JM. Measuring hospital adverse events: assessing inter-rater reliability and trigger performance of the global trigger tool. Int J Qual Health Care 22: 266–274. 41 Matlow AG, Cronin CMG, Flintoft V, Nijssen-Jordan C, Fleming M, Brady-Fryer B, et al. Description of the development and validation of the Canadian paediatric trigger tool. BMJ Qual Saf 2011; 20: 1–8. 42 Wiktorowicz M, Lexchin J, Paterson M, Mintzes B, Metge C, Light D, et al. In: Research networks involved in post-market pharmacosurveillance in the United States, United Kingdom, France, New Zealand, Australia, Norway and European Union: Lessons for Canada, ed the Canadian Institute for Patient Safety Edmonton, Canada: The Canadian Institute for Patient Safety, 2008. 43 Franklin BD, Birch S, Schachter M, Barbera N. Testing a trigger tool as a method of detecting harm from medication errors in a UK hospital: a pilot study. Int J Pharm Pract 2010; 18: 305–311. 44 Hwang J-I, Chin HJ, Chang Y-S. Characteristics associated with the occurrence of adverse events: A retrospective medical record review using the global trigger tool in a fully digitalized teaching care hospital in Korea. J Eval Clin Pract 2014; 20: 27–35. 45 Brenner S, Detz A, Lopez A, Horton C, Sarkar U. Signal and noise: applying a laboratory trigger tool to identify adverse drug events among primary care patients. BMJ Qual Saf 2012; 21: 670–675. 46 Kennerly DA, Saldan M, Kudyakov R, Da Graca B, Nicewander D, Compton J. Description and evaluation of adaptations to the
Performance of trigger tools in identifying adverse drug events
global trigger tool to enhance value to adverse event reduction efforts. J Patient Saf 2013; 9: 87–95. 47 Takata C, Mason W, Taketomo C, Logsdon T, Sharek PJ. Development, testing, and findings of a pediatric-focused trigger tool to identify medication-related harm in US children’s hospitals. Pediatrics 2008; 121: e927–e935. 48 Mattsson TO, Knudsen JL, Lauritsen J, Brixen K, Herrstedt J. Assessment of the global trigger tool to measure, monitor and evaluate patient safety in cancer patients: reliability concerns are raised. BMJ Qual Saf 2013; 22: 571–579. 49 Jhung MA, Budnitz DS, Mendelsohn AB, Weidenbach KN, Nelson TD, Pollock DA. Evaluation and overview of the National Electronic Injury Surveillance System-Cooperative Adverse Drug Event Surveillance Project (NEISS-CADES). Med Care 2007; 45: S96–S102. 50 Landrigan CP, Parry GJ, Bones CB, Hackbarth AD, Goldmann DA, Sharek PJ. Temporal trends in rates of patient harm resulting from medical care. N Engl J Med 2010; 363: 2124–2134.
Supporting Information Additional Supporting Information may be found in the online version of this article at the publisher’s web-site:
Appendix S1 Patient enrolment algorithm. We used a systematic selection algorithm to ensure enrolment of a representative sample during prospective data collection [21]. Appendix S2 Trigger tools are composed of individual triggers used to screen medical records. We evaluated all individual triggers that could plausibly be used to identify adverse drug events in adults and omitted those triggers intended to identify events unrelated to medication use (e.g. intensive care unit acquired pneumonia). Triggers consisting of serum drug concentrations or antidotes to medications that are not used in the outpatient setting were excluded (e.g. protamine). Orders to check the levels of the following medications are not common in the emergency department and were omitted from this study: ciclosporin [26], phenobarbital [26], procainamide [26], quinidine [26], theophylline [18, 26], lidocaine [18], gentamicin [18], tobramycin [18] and amikacin [18]. Letters indicating complaints from family members could not be obtained for our chart review and were therefore excluded from the study [1]. Appendix S3 Presence of 64 triggers in the records of patients, by adverse drug event status. Appendix S4 Study flow diagram.
http://onlinelibrary.wiley.com/doi/10.1111/bcp.13032/suppinfo.
Br J Clin Pharmacol (2016) 82 1048–1057
1057
