Research
Identifying Algorithms to Improve the Accuracy of Unverified Diagnosis Codes for Birth Defects
Public Health Reports Vol. XX(X) 1-8 ª 2018, Association of Schools and Programs of Public Health All rights reserved. Reprints and permission: sagepub.com/journalsPermissions.nav DOI: 10.1177/0033354918763168 journals.sagepub.com/home/phr
Jason L. Salemi, PhD, MPH1,2, Rachel E. Rutkowski, MSPH2, Jean Paul Tanner, PhD2, Jennifer L. Matas, BS1, and Russell S. Kirby, PhD2
Abstract Objectives: We identified algorithms to improve the accuracy of passive surveillance programs for birth defects that rely on administrative diagnosis codes for case ascertainment and in situations where case confirmation via medical record review is not possible or is resource prohibitive. Methods: We linked data from the 2009-2011 Florida Birth Defects Registry, a statewide, multisource, passive surveillance program, to an enhanced surveillance database with selected cases confirmed through medical record review. For each of 13 birth defects, we calculated the positive predictive value (PPV) to compare the accuracy of 4 algorithms that varied case definitions based on the number of diagnoses, medical encounters, and data sources in which the birth defect was identified. We also assessed the degree to which accuracy-improving algorithms would affect the Florida Birth Defects Registry’s completeness of ascertainment. Results: The PPV generated by using the original Florida Birth Defects Registry case definition (ie, suspected cases confirmed by medical record review) was 94.2%. More restrictive case definition algorithms increased the PPV to between 97.5% (identified by 1 or more codes/encounters in 1 data source) and 99.2% (identified in >1 data source). Although PPVs varied by birth defect, alternative algorithms increased accuracy for all birth defects; however, alternative algorithms also resulted in failing to ascertain 58.3% to 81.9% of cases. Conclusions: We found that surveillance programs that rely on unverified diagnosis codes can use algorithms to dramatically increase the accuracy of case finding, without having to review medical records. This can be important for etiologic studies. However, the use of increasingly restrictive case definition algorithms led to a decrease in completeness and the disproportionate exclusion of less severe cases, which could limit the widespread use of these approaches. Keywords birth defects, congenital malformations, accuracy, surveillance, positive predictive value Florida has the second-largest population-based birth defects surveillance system in the United States that relies predominantly on passive case ascertainment. Operated by the Florida Department of Health and a consortium of partners, the Florida Birth Defects Registry (FBDR) has monitored a population of more than 3 million live births since January 1, 1998, to detect structural, functional, and biochemical abnormalities in infants. Data from the FBDR and other state-based surveillance programs are used to support local investigations, epidemiologic studies, 1-4 health outcomes research,5-8 and national collaborative projects.9,10 However, for passive programs, in which cases are defined by unverified administrative diagnosis codes present in hospital discharge or service-related databases, the utility of surveillance data is hampered by the likelihood of under-ascertainment and reporting errors.11-15
Previous analyses have suggested that International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis codes,16 upon which many birth defect diagnoses are established, capture the general occurrence of a birth defect well but often fail to identify the birth 1
Department of Family and Community Medicine, Baylor College of Medicine, Houston, TX, USA 2 Birth Defects Surveillance Program, Department of Community and Family Health, College of Public Health, University of South Florida, Tampa, FL, USA Corresponding Author: Jason L. Salemi, PhD, MPH, Baylor College of Medicine, Department of Family and Community Medicine, 3701 Kirby Dr, BCM700, Houston, TX 77098, USA. Email:
[email protected]
2 defect with high accuracy.17 For example, the overall accuracy of the FBDR, estimated by using the positive predictive value (PPV) for 13 selected birth defects during 2007-2011, was 93.3%; that is, 6.7% of cases identified by the ICD-9CM codes were deemed to be false positives when validated by a diagnostic gold standard of medical record review with evidence of clinical confirmation of each birth defect. Despite a relatively high overall PPV, variation by birth defect ranged from 96.0% for gastroschisis to 1 Data Source
Abbreviations: FBDR, Florida Birth Defects Registry; PPV, positive predictive value. a Results are presented at the birth defect level. Because infants may have more than 1 birth defect included in the case confirmation project, the sum of individual birth defects in any birth defect group and overall may add to more than group/overall totals. Ordering of birth defects in the individual birth defects group is alphabetical. b Case confirmation protocol included medical record review by a trained abstractor followed by documentation of each confirmed birth defect using modified British Pediatric Association diagnosis codes.21 c Calculated as (number of FBDR cases confirmed with the same birth defect / number of cases of the birth defect reported by the FBDR) 100.
Individual birth defects Anencephaly Cleft lip without cleft palate Cleft palate without cleft lip Cleft palate with cleft lip Gastroschisis Hypoplastic left heart syndrome Hypospadias Reduction deformities, upper limb Reduction deformities, lower limb Spina bifida without anencephaly Tetralogy of Fallot Transposition of the great arteries Trisomy 21 (Down syndrome) Birth defect groups Any orofacial cleft Any included congenital heart defect Any limb reduction deformity Any included birth defect
Birth Defecta,b
Birth DefectSpecific No. of Cases No. of Cases Reported by PPVc of FBDR Reported by FBDR FBDR (95% CI)
Original Case Definition
Birth Defect Identified by Multiple Codes or Multiple Encounters
Table 2. Birth defect-specific PPV of the FBDR for selected birth defects using various passive ascertainment algorithms to identify cases, 2009-2011
6 17 65 160 150 82 115 479 10 26 107 189 148 407 456 52 411 1988
39 162 367 267 300 185 1999 85 129 173 316 286 776 722 195 702 4772
No. of Cases Reported by FBDR
36.8 73.3 41.5 58.3
56.4 59.9 56.4 43.8 72.7 37.8 76.0 88.2 79.8 38.2 40.2 48.3 47.6
% Decrease in No. of Cases
332 43 302 1123
2 43 123 112 68 68 89 6 21 63 136 122 244
No. of Cases Reported by FBDR
54.0 77.9 57.0 76.5
94.9 73.5 66.5 58.1 77.3 63.2 95.5 92.9 83.7 63.6 57.0 57.3 68.6
% Decrease in No. of Cases
Birth Defect Identified by Multiple Codes or Multiple Encounters, but From Only 1 Data Source
124 9 109 865
15 22 37 38 14 47 390 4 5 44 53 26 163
No. of Cases Reported by FBDR
82.8 95.4 84.5 81.9
61.5 86.4 89.9 85.8 95.3 74.6 80.5 95.3 96.1 74.6 83.2 90.9 79.0
% Decrease in No. of Cases
Birth Defect Identified by >1 Data Source
Abbreviation: FBDR, Florida Birth Defects Registry. a Results are presented at the birth defect level. Because infants may have more than 1 birth defect included in the case confirmation project, the sum of individual birth defects in any birth defect group and overall may add to more than group/overall totals. Ordering of birth defects in the individual birth defects group is alphabetical. b Case confirmation protocol included medical record review by a trained abstractor followed by documentation of each confirmed birth defect using modified British Pediatric Association diagnosis codes.21
Individual birth defects Anencephaly Cleft lip without cleft palate Cleft palate without cleft lip Cleft palate with cleft lip Gastroschisis Hypoplastic left heart syndrome Hypospadias Reduction deformities, lower limb Reduction deformities, upper limb Spina bifida without anencephaly Tetralogy of Fallot Transposition of the great arteries Trisomy 21 (Down syndrome) Birth defect groups Any included congenital heart defect Any limb reduction deformity Any orofacial cleft Any included birth defect
Birth Defecta,b
Original Case Definition, No. of Cases Reported by FBDR
Birth Defect Identified by Multiple Codes or Multiple Encounters
Table 3. Birth defect-specific completeness of ascertainment (relative to the original FBDR case definition) for selected birth defects using various passive ascertainment algorithms to identify cases, 2009-2011
Salemi et al Approximately 60% of birth defects surveillance programs in the United States rely predominantly or exclusively on a passive strategy to ascertain cases,22 as do many other disease registries. Our results should encourage programs with limited resources for case verification to explore strategies to maximize various quality metrics (eg, accuracy, completeness, timeliness) for case ascertainment in accordance with programmatic goals. Our findings suggest that if a passive surveillance program were contributing data to a national study on the etiology of birth defects and the inclusion of false-positive diagnoses were a substantial threat to internal validity, the false-positive rate for selected birth defects could be minimized without medical record review by using alternative case definition algorithms. However, depending on the definition used, the cases selected for such a study, although being true cases, may not be representative of all cases. For example, in our study, the PPV for the ICD9-CM code for hypoplastic left heart syndrome (746.7) increased from 77.8% to 100.0% by requiring the code to appear in more than 1 data source. Infants whose diagnosis of hypoplastic left heart syndrome was documented in more than 1 data source had more severe complications that required more inpatient hospitalizations, outpatient surgeries, and emergency department visits than infants whose diagnosis of the syndrome was documented in only 1 data source. Thus, any study based on this subset of more severe cases may overestimate morbidity, mortality, costs, and epidemiologic associations, because the case definition requires more frequent contact with the medical system. Conversely, for fundamental activities such as routine monitoring, aimed at detecting changes in the environment or health status of populations or for planning and resource allocation, overly restrictive case definition algorithms are not recommended. They are not recommended because the limitations of the excessively low completeness of ascertainment outweigh the benefits of smaller-scale increases in accuracy. For example, in our study, requiring the ICD-9-CM code for Down syndrome (758.0) to appear in more than 1 medical encounter (ie, >1 discharge record) improved PPV from 93.3% to 99.6%, but such a case definition would capture barely half of all confirmed cases and substantially underestimate the prevalence and burden of cases. Therefore, when exploring alternative algorithms for capturing data on birth defects or any other health outcome, decision makers should consider these quality metric tradeoffs, including timeliness,24 so that adopted strategies align with intended goals.
Limitations This study had several limitations. First, despite considering algorithms that evaluate plausible case definitions based on diagnoses appearing in multiple ICD codes, medical encounters, and data sources, we did not explore all possible definitions. For example, Smith et al25 found that including in their algorithm variables related to demographic characteristics and specialty providers in addition to the number of
7 times a diagnosis code was used improved the PPV of a passive surveillance system for muscular dystrophy. Second, although the FBDR links to a broad range of data sources to establish an annual cohort of infants born with birth defects, the FBDR does not link to educational services, government insurance data, or data sources that contain information on terminations or stillbirths. These data sources might capture data on cases that would otherwise have been missed or may not have multiple occurrences in hospital administrative data sources. The availability of these data sources alone may have improved the accuracy and completeness beyond that attainable by varying case definition algorithms based on current FBDR data sources. Third, the FBDR does not use medical specialists to review records. A pediatric cardiologist, for example, does not review the entire chart of a child who receives a diagnosis of hypoplastic left heart syndrome, and in some registries, approximately 35% of hypoplastic left heart syndrome cases are misdiagnosed.26 Despite these limitations, this study points to some advantages. The data source was large and statewide, providing case information on a diverse population of nearly 5000 children with birth defects of major public health importance. Unlike other passive registries, the FBDR has an enhanced surveillance component that allows for medical record review to verify selected birth defects, providing a gold standard diagnosis upon which an evaluation of the accuracy and completeness of the passive statewide program could be based. The ultimate goal of this study was to determine whether the accuracy of a passive surveillance program could be improved, specifically in situations in which case confirmation via medical record review either would not be possible or would be resource prohibitive. Our results suggest that excellent accuracy of ICD codes for selected birth defects can be achieved but that decreases in completeness and generalizability of cases are inevitable if increasingly restrictive case definition algorithms are applied.
Conclusions Surveillance programs and disease registries use their data for various purposes, including incidence and/or prevalence estimation, temporal trends analysis, local investigations, epidemiologic association studies, health outcomes research, and etiological examinations.27 Moreover, for registries based on rare outcomes, regional or national collaborative projects based on pooled data may be necessary to achieve sufficient statistical precision to contribute meaningfully to the evidence base. Improving the diagnostic accuracy and completeness of registries using passive case-finding strategies is essential to their inclusion in pooled data sources. The methods described in this study have broader applicability to public health surveillance and research using plan- or population-based claims data as well as administrative public health data collected from hospitals for conditions that require multiple health encounters.
8 Declaration of Conflicting Interests The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The analysis for this project was supported by an award from the Centers for Disease Control and Prevention (CDC) (Grant No. 1 NU50DD004946-01-00). The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of CDC, the Florida Department of Health, the University of South Florida, or Baylor College of Medicine.
References 1. Block SR, Watkins SM, Salemi JL, et al. Maternal prepregnancy body mass index and risk of selected birth defects: evidence of a dose-response relationship. Paediatr Perinat Epidemiol. 2013;27(6):521-531. 2. Nembhard WN, Salemi JL, Hauser KW, Kornosky JL. Are there ethnic disparities in risk of preterm birth among infants born with congenital heart defects? Birth Defects Res A Clin Mol Teratol. 2007;79(11):754-764. 3. Kucik JE, Cassell CH, Alverson CJ, et al. Role of health insurance on the survival of infants with congenital heart defects. Am J Public Health. 2014;104(9):e62-e70. 4. Salemi JL, Pierre M, Tanner JP, et al. Maternal nativity as a risk factor for gastroschisis: a population-based study. Birth Defects Res A Clin Mol Teratol. 2009;85(11):890-896. 5. Dawson AL, Cassell CH, Oster ME, et al. Hospitalizations and associated costs in a population-based study of children with Down syndrome born in Florida. Birth Defects Res A Clin Mol Teratol. 2014;100(11):826-836. 6. Delmelle EM, Cassell CH, Dony C, et al. Modeling travel impedance to medical care for children with birth defects using geographic information systems. Birth Defects Res A Clin Mol Teratol. 2013;97(10):673-684. 7. Peterson C, Dawson A, Grosse SD, et al. Hospitalizations, costs, and mortality among infants with critical congenital heart disease: how important is timely detection? Birth Defects Res A Clin Mol Teratol. 2013;97(10):664-672. 8. Radcliff E, Cassell CH, Tanner JP, et al. Hospital use, associated costs, and payer status for infants born with spina bifida. Birth Defects Res A Clin Mol Teratol. 2012;94(12):1044-1053. 9. Wang Y, Liu G, Canfield MA, et al. Racial/ethnic differences in survival of United States children with birth defects: a population-based study. J Pediatr. 2015;166(4):819-826. 10. Canfield MA, Mai CT, Wang Y, et al. The association between race/ethnicity and major birth defects in the United States, 1999-2007. Am J Public Health. 2014;104(9):e14-e23. 11. Boulet SL, Correa-Villasenor A, Hsia J, Atrash H. Feasibility of using the National Hospital Discharge Survey to estimate the prevalence of selected birth defects. Birth Defects Res A Clin Mol Teratol. 2006;76(11):757-761.
Public Health Reports XX(X) 12. Callif-Daley FA, Huether CA, Edmonds LD. Evaluating false positives in two hospital discharge data sets of the Birth Defects Monitoring Program. Public Health Rep. 1995;110(2):154-160. 13. Cronk CE, Malloy ME, Pelech AN, et al. Completeness of state administrative databases for surveillance of congenital heart disease. Birth Defects Res A Clin Mol Teratol. 2003;67(9):597-603. 14. Frohnert BK, Lussky RC, Alms MA, Mendelsohn NJ, Symonik DM, Falken MC. Validity of hospital discharge data for identifying infants with cardiac defects. J Perinatol. 2005;25(11):737-742. 15. Hexter AC, Harris JA. Bias in congenital malformations information from the birth certificate. Teratology. 1991;44(2):177-180. 16. National Center for Health Statistics. International classification of diseases, ninth revision, clinical modification (ICD-9CM). https://www.cdc.gov/nchs/icd/icd9cm.htm. Published 2015. Accessed September 19, 2017. 17. Salemi JL, Tanner JP, Sampat D, et al. The accuracy of hospital discharge diagnosis codes for major birth defects: evaluation of a statewide registry with passive case ascertainment. J Public Health Manag Pract. 2016;22(3):E9-E19. 18. Salemi JL, Tanner JP, Bailey M, Mbah AK, Salihu HM. Creation and evaluation of a multi-layered maternal and child health database for comparative effectiveness research. J Registry Manag. 2013;40(1):14-28. 19. Salemi JL, Tanner JP, Block S, et al. The relative contribution of data sources to a birth defects registry utilizing passive multisource ascertainment methods: does a smaller birth defects case ascertainment net lead to overall or disproportionate loss? J Registry Manag. 2011;38(1):30-38. 20. National Center for Health Statistics. International classification of diseases, tenth revision, clinical modification (ICD-10CM). https://www.cdc.gov/nchs/icd/icd10cm.htm. Published 2017. Accessed September 19, 2017. 21. Centers for Disease Control and Prevention. Birth defects and genetic diseases branch 6-digit code: for reportable congenital anomalies. https://www.cdc.gov/ncbddd/birthdefects/docu ments/MACDPcode0807.pdf. Published 2007. Accessed November 3, 2017. 22. Nembhard WN, Wang T, Loscalzo ML, Salemi JL. Variation in the prevalence of congenital heart defects by maternal race/ ethnicity and infant sex. J Pediatr. 2010;156(2):259-264. 23. SAS Institute, Inc. SAS Version 9.4 for Windows. Cary, NC: SAS Institute, Inc; 2016. 24. Salemi JL, Tanner JP, Anjohrin SB, et al. Evaluating difficult decisions in public health surveillance: striking the right balance between timeliness and completeness. J Registry Manag. 2015;42(2):48-61. 25. Smith MG, Royer J, Mann JR, McDermott S. Using administrative data to ascertain true cases of muscular dystrophy: rare disease surveillance. JMIR Public Health Surveill. 2017;3(1):e2. 26. Hirsch JC, Copeland G, Donohue JE, Kirby RS, Grigorescu V, Gurney JG. Population-based analysis of survival for hypoplastic left heart syndrome. J Pediatr. 2011;159(1):57-63. 27. Anderka M, Mai CT, Romitti PA, et al. Development and implementation of the first national data quality standards for population-based birth defects surveillance programs in the United States. BMC Public Health. 2015;15:925.
