SPECIAL FEATURES
Detecting medication errors
S P E C I A L F E AT U R E
Observation method of detecting medication errors KENNETH N. BARKER, ELIZABETH A. FLYNN, AND GINETTE A. PEPPER Am J Health-Syst Pharm. 2002; 59:2314-6
O
bservation as a method of detecting and counting errors in drug administration dates back to the early 1960s.1 It is one of the four basic methods of collecting data, with the others being self-reports (e.g., interviews and questionnaires), testing, and physical evidence (e.g., document review).2 Researchers have understood since the beginning of medication error research that no single method of error detection will work in all situations. The special requirements necessary for the observation method include events that are visible, predictable, and of limited duration.
Validity issues. Validity is a fundamental component of any research initiative, and a common philosophy of researchers is that “if you do not have validity, you do not have anything.” The observation method offers the following advantages in all areas of research: • Knowledge of errors by subjects is not required, • Willingness to report is not required, • Remembering is not required, • The ability to communicate is not required, • Selective perception of subjects is unrelated,
KENNETH N. BARKER, PH.D., is Director, and ELIZABETH A. FLYNN, PH.D., is Associate Research Professor, Center for Pharmacy Operations and Designs, Auburn University, Auburn, AL. GINETTE A. PEPPER, PH.D., FAAN, is Associate Professor and Director, Nursing Doctorate Program (Interim), University of Colorado Health Sciences Center, Denver. Address correspondence to Dr. Barker at the Center for Pharmacy Operations and Designs, 128 Miller Hall, Auburn University, Au-
2314
Am J Health-Syst Pharm—Vol 59 Dec 1, 2002
• Observer inference is involved, and • The effect of the observer on the observed is not significant.
The last item is a key issue, and it was rated not significant only because there are ways to make it so. Ways of protecting the validity of the observation method against observer bias include using experienced observers, such as pharmacists and nurses, and training the observers to be objective, unobtrusive, and nonjudgemental. History of the observation method. Key studies of the observation technique for detecting medication errors have appeared in the litera-
burn, AL 36849-5506 (
[email protected]). Supported by the Latiolais Leadership Program and the ASHP Research and Education Foundation. Presented at the Measuring Medication Safety in Hospitals Conference, Tucson, AZ, April 8–9, 2002. Copyright © 2002, American Society of Health-System Pharmacists, Inc. All rights reserved. 1079-2082/02/1201-2314$06.00.
SPECIAL FEATURES Detecting medication errors
ture. The first exploratory study, which monitored nine nurses over two days, confirmed the existence and nature of errors.1 A second descriptive study outlined the frequency and distribution of errors and provided clues to the causes of errors.3 This study followed 32 nurses each for five successive days. A unit dose study conducted by Barker et al.4 in 1969 measured the effectiveness of an intervention and involved the observation of 192 nurse-days before and 64 nurse-days after the installation of a unit dose system. The author and his colleagues subsequently decided that focusing on the testing of promising interventions produces faster results (reducing error rates) than rootcause analysis. In 1984, the Health Care Financing Administration (now called the Centers for Medicare and Medicaid Services [CMS]) adopted the observation method in its annual surveys of all long-term-care facilities in the United States. CMS mandated that facilities have error rates of 5% or below or no significant errors to be eligible for reimbursement under Medicare and Medicaid. Since that time, a wealth of experience on the observation method has accumulated; more than 40 follow-up studies in several countries have appeared since 1984.5 Evaluating the observation method. Several studies have evaluated the observation method itself. In 1962, Barker and McConnell1 compared the observation method with other strategies extrapolated over one year: Incident reports uncovered 36 events, anonymous questionnaires 10 events, and observation 1424. Interobserver agreement (i.e., two observers’ views of the same event) has also been studied extensively. Barker et al.3 were able to demonstrate a high level of agreement, but in the study conducted for CMS in 1984, the level of agreement was less clear.6 The observation technique produces
results that are significantly more valid and reliable than the selfreporting methods that have been examined to date. The clinical significance of errors uncovered by the observation technique has been studied extensively.5 Clinical significance has been defined as an injury actually suffered by a particular patient (i.e., an adverse drug event).7 Barker and his colleagues have taken a slightly different approach, defining clinical significance as those doses that put a typical patient at risk for an adverse event and looking at ways of avoiding situations that put patients at risk. Years of experience with the observation technique have revealed many practical advantages over other methods: • The method is easily understood, • Data are easy to use for identifying trends and benchmarking, • Data are available within hours, • The method is systems oriented and views errors in doses as defects in a system, • The method is objective and does not assign blame, • The method is defensible, with all doses being examined and errors witnessed, • The method enables problem-based continuing education that focuses on “our” errors, • The method facilitates evidencebased testing that can evaluate proposed system changes, • Quality can be measured quantitatively by third parties, and • The method establishes track records for national use, such as those used by CMS in long-term-care facilities.
Perhaps the best example of the impact of the observation method was in evaluating the unit dose concept, which was eventually adopted by about 90% of U.S. hospitals after a series of studies involving observation.8 Comparison of methods for detecting medication errors. Flynn et
al.9 reported the results of a recent study that compared methods for detecting medication errors in 36 hospitals and skilled-nursing facilities. Three methods for detecting medication errors—incident reports, chart review, and observation—were used, along with three types of data collectors (registered nurses, licensed practical nurses, and pharmacy technicians). The study was conducted from May to November 1999 at 18 randomly selected facilities in Denver and 18 in Atlanta, including accredited and nonaccredited facilities. A medication administration error was defined as any difference between what the patient received or was supposed to receive and what the prescriber intended in the original order. This is an important difference between this study and previous ones that have categorized errors as those either having the potential to harm patients or leading to actual harm. The error categories were those typically included in such studies (e.g., unauthorized drug, omission, extra dose, wrong dose, wrong dosage form, wrong route of administration, wrong time, and wrong technique). The comparison doses were those that were administered and observed by the data collectors or those that were supposed to have been given during the observation period. A head-to-head comparison of each of the methods for detecting errors was performed on the same doses that were administered during the observation period. The cost of each method was determined by calculating the time it took each data collector to detect errors with each of the three methods. Each data collector was randomly assigned to use one of the methods. For instance, an investigator assigned to observe administration would watch the administration of about 50 doses on a nursing unit and calculate an error rate. The chart review method was based on the technique of
Am J Health-Syst Pharm—Vol 59 Dec 1, 2002
2315
SPECIAL FEATURES
Detecting medication errors
Bates et al.10 in which drug orders were compared with the medication administration record. A chart reviewer would review the charts of up to 10 patients involved in the observation and look for other clues in the chart, such as “red-flag” drug orders (e.g, stat orders for insulin), results of laboratory tests (e.g., glucose concentration, >400 mg/dL), and significant physician and nurse progress notes, and record discrepancies on a form. Incident reports were reviewed two weeks later to allow time for the errors to be filed. Originals and photocopies of the incident reports were reviewed, and the errors were evaluated and categorized. The date and time of each error were also noted. At each facility, a research pharmacist performed all three methods in a blinded fashion. The potential of each error for harm was determined by a panel of three physicians at the Brigham and Women’s Hospital in Boston. The panel used the CMS guidelines for determining the clinical significance of an error, as well as some patient information provided by the pharmacist who reviewed the chart. Two of the three physicians had to rate the error as significant for it to be counted as such. A total of 2557 doses were observed during the seven-month study period. A total of 456 errors were detected by pharmacists doing the double-check of the observation method, 34 were detected by chart review, and 1 was detected via incident report (one of the same errors detected by observation). There were some interesting discrepancies between the evidence collected by chart review and that collected by observation. In one instance, the chart review identified an omission of a dose of docusate sodium 100 mg, but the observer recorded that the patient had refused the drug. On another occasion, the chart review indicated a dose of fluoxetine hydrochloride 40 mg as an omission, but the observer had witnessed administration of the dose.
2316
The evidence supported the conclusion that the most efficient and accurate method of collecting error data was the observation method. Modifications of the observation method. Nurses have questioned whether the large numbers of errors detected in observational studies are believable. One of the authors (G.A.P., a nurse) is involved in an ongoing research project that will evaluate the effect of nurses’ working conditions on medication safety, with the goal of developing a hierarchical linear model of system factors that affect the safety of medication use. It is well recognized that research into the causes of medication errors is still needed, and the observation method may be the most sensitive and valid method for identifying the causes of medication errors. The study is being funded by the Agency for Healthcare Research and Quality for three years and will evaluate a national sample of 240 nurses from 60 units in 12 hospitals. The study will look at factors associated with the hospital, nursing units (e.g., staffing and types of pharmaceutical services), nurses (e.g., factors that are modifiable by hospital policy, such as number of hours worked), episodes (i.e., all doses that are delivered to one patient at one time), and doses. The episode and dosage factors are those for which the observation method can be expanded and additional data about the causes or correlates of errors collected. Nurses have dual roles in medication safety in that they are both producers of errors and defenders against them. As a result, the study team plans to measure both accuracy and error rates and to take into account actions taken by nurses to prevent adverse drug events. They will collect data on observable factors that may contribute to errors, such as adherence to standards and best practices, and environmental hindrances, such as interruptions, distractions, light, noise, and availabili-
Am J Health-Syst Pharm—Vol 59 Dec 1, 2002
ty of drugs, equipment, and patients. To minimize the time and cost associated with the traditional observation method, a computerized record on a handheld computer that has a touchscreen and handwriting recognition and audiorecording capabilities is being developed. Data will be collected on drug, formulation, strength, number, and route as they appear during drug administration, on the medication administration record, and in the physician’s orders on the patient chart. The pilot study will test the reliability and validity of the observation method. Nurses will be able to provide feedback on the obtrusiveness of the method. Conclusion. The observation method has been used to detect medication errors since the early 1960s, and there is evidence that it is valid, efficient, and accurate. References 1. Barker KN, McConnell WE. The problems of detecting medication errors in hospitals. Am J Hosp Pharm. 1962; 19:360-9. 2. Barker KN. Data collection techniques: observation. Am J Hosp Pharm. 1980; 37: 1235-43. 3. Barker KN, Kimbrough WW, Heller WM. A study of medication errors in a hospital. Fayetteville, AR: Univ. of Arkansas; 1966. 4. Barker KN. The effects of an experimental medication system on medication errors and costs: part one. Introduction and errors study. Am J Hosp Pharm. 1969; 26:324-33. 5. Allan EL, Barker KN. Fundamentals of medication errors research. Am J Hosp Pharm. 1990; 47:555-71. 6. Barker KN, Mikeal RL, Pearson RE et al. Medication errors in nursing homes and small hospitals. Am J Hosp Pharm. 1982; 39:987-91. 7. Leape LL, Bates DW, Cullen DJ et al. Systems analysis of adverse drug events. JAMA. 1995; 274:35-43. 8. Dean BS, Allan EL, Barber ND et al. Comparison of medication errors in an American and a British hospital. Am J Health-Syst Pharm. 1995; 52:2543-9. 9. Flynn EA, Barker KN, Pepper GA et al. Comparison of methods for detecting medication errors in 36 hospitals and skilled-nursing facilities. Am J HealthSyst Pharm. 2002; 59:436-45. 10. Bates DW, Cullen DJ, Laird N et al. Incidence of adverse drug events and potential adverse drug events. JAMA. 1995; 274:29-34.
