American Journal of Epidemiology Copyright ª 2007 by the Johns Hopkins Bloomberg School of Public Health All rights reserved; printed in U.S.A.
Vol. 165, No. 12 DOI: 10.1093/aje/kwm034 Advance Access publication April 3, 2007
Practice of Epidemiology Using Probabilistic Corrections to Account for Abstractor Agreement in Medical Record Reviews
Timothy L. Lash1,2, Matthew P. Fox1, Soe Soe Thwin1,3, Ann M. Geiger4,5, Diana S. M. Buist6, Feifei Wei7, Terry S. Field8,9, Marianne Ulcickas Yood10,11, Floyd J. Frost12, Virginia P. Quinn5, Marianne N. Prout3, and Rebecca A. Silliman1,2,3 7
2
8
Boston University School of Public Health, Boston, MA. Boston University School of Medicine, Boston, MA. 3 Boston Medical Center, Boston, MA. 4 Wake Forest University School of Medicine, WinstonSalem, NC. 5 Kaiser Permanente Southern California, Los Angeles, CA. 6 Group Health Center for Health Studies, Seattle, WA.
HealthPartners Research Foundation, Minneapolis, MN. University of Massachusetts Medical School, Worcester, MA. 9 Fallon Community Health Plan, Worcester, MA. 10 Henry Ford Health System, Detroit, MI. 11 Yale University School of Medicine, New Haven, CT. 12 Lovelace Health Systems, Albuquerque, NM.
Received for publication June 14, 2006; accepted for publication December 6, 2006.
The quality of medical record abstracts is often characterized in a reliability substudy. These results usually indicate agreement, but not the extent to which lack of agreement affects associations observed in the complete data. In this study, medical records were reviewed and abstracted for patients diagnosed with stage I or stage II breast cancer between 1990 and 1994 at one of six US Cancer Research Network sites. For a subsample, interrater reliability data were available. The authors calculated conventional hazard ratios and 95% confidence intervals for the association of demographic, tumor, and treatment characteristics with recurrence rate. These conventional estimates of effect were compared with three sets of estimates and 95% simulation intervals that took account of the uncertainty assessed by lack of agreement in the reliability substudy. The rate of recurrence was associated with increasing cancer stage and with treatment modality but not with demographic characteristics. The hazard ratios and simulation intervals that took account of the reliability data showed that the simulation interval grew wider as the sources of uncertainty taken into account grew more complete, but the associations expected a priori remained readily apparent. While many investigators use reliability data only as a metric for data quality, a more thorough approach can also quantitatively depict the uncertainty in the observed associations. breast neoplasms; data collection; epidemiologic methods; medical records
Abbreviations: CCI, Charlson comorbidity index; IRR, interrater reliability.
Medical record review provides data with which to confirm subject eligibility, ascertain disease outcomes, or characterize disease severity, treatments received, or comorbid conditions (1). Abstractors review medical records to collect requisite data using a standardized form accompanied by a code book to assure uniform coding decisions. With the advent of electronic medical record data, some data fields
can be completed through direct linkage of electronic medical records with a data collection system (2). Nonetheless, many data fields still require review of multiple data sources or free text to abstract the information. Abstractors should receive uniform training (3, 4), including an explanation of the data collection form (4) and its code book and pilot practice with a sample similar to the
Correspondence to Dr. Timothy L. Lash, Department of Epidemiology, Boston University School of Public Health, 715 Albany Street, TE3, Boston, MA 02118 (e-mail:
[email protected]).
1454
Am J Epidemiol 2007;165:1454–1461
Downloaded from http://aje.oxfordjournals.org/ by guest on December 22, 2015
1
Using Probabilistic Corrections for Abstractor Agreement
MATERIALS AND METHODS
The parent study enrolled 1,859 women aged 65 years or older with early-stage breast cancer who received health care in one of six geographically dispersed community-based integrated health-care systems (8). The study aims were to identify patient and tumor characteristics associated with receipt of treatment and adverse breast cancer outcomes. The study protocol was approved by institutional review boards at the coordinating center and the enrollment sites. Population
We identified potentially eligible women diagnosed with American Joint Committee on Cancer (9) stage I, IIA, or IIB breast cancer in one of six health-care delivery systems participating in the National Cancer Institute-funded Cancer Research Network (Group Health Center for Health Studies, Seattle, Washington; Kaiser Permanente Southern California, Pasadena, California; Lovelace Health Systems, Albuquerque, New Mexico; Henry Ford Health System, Detroit, Michigan; HealthPartners, Minneapolis, Minnesota; and Fallon Community Health Plan, Worcester, Massachusetts). Eligible patients had a histologically confirmed first breast neoplasm diagnosed between 1990 and 1994 and had been enrolled in their health plan for at least 12 months before and after their diagnosis, unless they died within the first year after diagnosis. We excluded women with other maligAm J Epidemiol 2007;165:1454–1461
nancies, except nonmelanoma skin cancer, diagnosed within 5 years before or 30 days after the breast cancer diagnosis and women with bilateral breast cancer. For this analysis, we restricted the sample to women who received either breast-conserving surgery or mastectomy. Data collection
We collected demographic, tumor, treatment, comorbidity, and follow-up data from medical records and electronic data sources, including cancer registry, administrative, and clinical databases. We initially populated the database with electronically available data and verified all preloaded data by medical record review, except cancer registry data elements reported to the National Cancer Institute’s Surveillance, Epidemiology, and End Results registry (10). One person used a standard procedure to train all medical record abstractors at all sites. Standardized medical record reviews were conducted on-site by the trained abstractors, and the data were entered directly into a computer-based data collection system. Analytic variables Breast cancer recurrence. We defined breast cancer recurrence as invasive cancer diagnosed in the same breast, in the lymph nodes, or at a distant site at least 120 days after the original diagnosis or after completion of the last surgery in the first course of treatment, whichever was later. We followed women to the first of the following: date of disenrollment, date of recurrence, date of death, or 10 years from the date of diagnosis. Demographic data. We gathered information on each woman’s date of birth from cancer registry databases for sites with cancer registries (Group Health, Kaiser Permanente, Lovelace, Henry Ford) and from the women’s medical records at the sites without cancer registries (Fallon, HealthPartners). We classified women into age groups of 65–
