AJCP / ORIGINAL ARTICLE
Data-Driven Iterative Refinement of Bone Marrow Testing Protocols Leads to Progressive Improvement in Cytogenetic and Molecular Test Utilization Adam C. Seegmiller, MD, PhD, Annette S. Kim, MD, PhD, Claudio A. Mosse, MD, PhD, Aaron C. Shaver, MD, PhD, Mary Ann Thompson, MD, PhD, Shaoying Li, MD, David R. Head, MD, and Mary M. Zutter, MD From the Department of Pathology, Microbiology, and Immunology, Vanderbilt University School of Medicine, Nashville, TN.
Key Words: Bone marrow; Test utilization; Hematopathology Am J Clin Pathol November 2016;146:585-593 DOI: 10.1093/AJCP/AQW180
ABSTRACT Objectives: To determine the effect of iterative refinement of standard ordering protocols on test utilization and results for bone marrow biopsy specimens. Methods: Eighteen months of test utilization and result data were used to revise the protocols that determine cytogenetic and molecular test selection on bone marrow specimens and then compared with data obtained following protocol revision. Results: Revision of protocols resulted in reduction in total tests and associated charges, due to a decrease in tests both concordant and discordant with the protocols. These reductions only occurred in diseases for which revisions were made and were limited to cases in which reflex testing was performed. There was an increase in the fraction of positive tests, which was also limited to reflex testing. Conclusions: Data-driven iterative revision of protocols further improves test utilization and performance, while reducing cost. Analysis of testing data can be used to continuously improve test ordering decisions.
© American Society for Clinical Pathology, 2016. All rights reserved. For permissions, please e-mail:
[email protected] 585
A number of recent initiatives have drawn attention to the challenge of inappropriate laboratory testing.1,2 A recent meta-analysis3 suggests that inappropriate laboratory test utilization, including both overutilization and underutilization, remains a significant issue. This is particularly true in the area of cancer diagnostics.4,5 Many different methods have been recommended to address this issue.6 At our institution, we combined several of these approaches in the development of a multidisciplinary group termed the hematopathology diagnostic management team (DMT). This group of pathologists and clinical hematologists worked together to develop standard ordering protocols (SOPs) defining which cytogenetic and molecular tests should be performed on bone marrow biopsy specimens based on the patient’s disease, stage of therapy, and prior test results. Application of these SOPs significantly reduced both overutilization and underutilization of tests, leading to decreased testing costs and improved test performance.7 Subsequent studies from other groups have also demonstrated that an algorithmic approach to hematopathology diagnostics can reduce test utilization.8,9 One of the weaknesses of this approach was the relative dearth of published evidence upon which these SOPs could be based. In many instances, the DMT had to rely on clinical judgment, experience, and intuition alone to guide decisions in the absence of formal evidence. Indeed, analysis of testing outcomes indicated that even with proper application of the SOPs, there were still test results that provided little or no helpful clinical information.7
Am J Clin Pathol 2016;146:585-593 585 DOI: 10.1093/ajcp/aqw180
Seegmiller et al / ITERATION
OF
BONE MARROW TESTING PROTOCOLS
To address this weakness, we applied the concept of a rapid learning system10,11 to the hematopathology DMT. Data on test ordering patterns and test outcomes were collected for 6 months prior to and 12 months after implementation of the DMT. These data were used to update, modify, and refine the SOPs based on the evidence collected. We hypothesized that this iterative approach would result in further improvement in test utilization. The current study compares the effect on test utilization, results, and costs of the initial SOPs with that of the revised SOPs. It demonstrates that evidence-based refinement of SOPs can further reduce unnecessary testing, decrease costs, and improve test performance.
Materials and Methods The Hematopathology DMT The structure and function of the Vanderbilt hematopathology DMT were described previously.7 Briefly, the DMT is composed of clinical hematologists, hematopathologists, and experts in biomedical informatics. They are tasked with developing SOPs, which are algorithms that define when particular ancillary cytogenetic and molecular tests should be ordered on bone marrow biopsy specimens. The SOPs are specific for disease, stage of therapy, and prior testing results. Typically, the SOPs are applied by the hematopathologist after preliminary examination of the specimen by microscopy and flow cytometry. However, the hematologists retain the ability to order tests a la carte, if clinically indicated. Evaluation and Revision of SOPs During the first year of DMT implementation, test ordering and result data were collected for all of the bone marrow biopsy specimens performed during that year and compared with similar data obtained from the previous 6 months.7 DMT members then evaluated the SOPs based on these data, as well as new or updated clinical guidelines, and the collective experience of the group in applying the SOPs to various clinical situations. Based on this evaluation, the SOPs were revised to reduce testing that the analysis showed to be unnecessary and to reinstate testing that had been removed in the prior SOP if the analysis indicated that it was clinically useful. In addition, the SOPs were adjusted to better reflect clinical practice and to ease interpretation, as well as to better adhere to published clinical guidelines. The resulting changes are detailed in Table 1 . These changes were reviewed, modified, and approved by all
586 Am J Clin Pathol 2016;146:585-593 586 DOI: 10.1093/ajcp/aqw180
members of the hematology and hematopathology teams at Vanderbilt.
Test Utilization Analysis This study was approved by the Institutional Review Board of Vanderbilt University. Data were collected for three testing periods: (1) 6 months prior to DMT implementation (pre-DMT; August 14, 2010, through February 13, 2011), (2) 12 months after DMT implementation (SOP-1; February 14, 2011, through February 13, 2012), and (3) 12 months after SOP revision (SOP-2; February 14, 2012, through February 13, 2013). Data for pre-DMT and SOP-1 were collected and reported previously.7 Laboratory systems and medical records were examined to determine the following for each bone marrow biopsy specimen: diagnosis and stage of therapy, number and type of cytogenetic and molecular tests performed, whether the tests were ordered by the pathologist following the DMT testing protocol (“reflex testing”) or by the hematologist (“nonreflex testing”), and whether the results were positive (abnormal) or negative (normal). In addition, the test orders were compared with the recommendations of the appropriate SOP and categorized as concordant (ordered as recommended), discordant (ordered but not recommended), or omitted (recommended but not ordered). The examiners were blinded as to whether the tests were ordered by the DMT reflex protocol or not. The data were collected into a REDCap database (http://redcap.van derbilt.edu).12 Financial analysis was performed from the perspective of payers as previously described.7 Charge per test was estimated as the mean reimbursement of each test to the Vanderbilt University Medical Center based on contracted rates for services from both public and private payers and taking into account the relative payer mix at Vanderbilt. The cost per test was held constant across the entire time period of the study to eliminate the effect of price fluctuations.
Statistical Analysis Statistical analysis was performed using Prism software (version 5.0; GraphPad Software, La Jolla, CA). Categorical variables were compared using the v2 test. Quantitative variables were compared across pre-DMT, SOP-1, and SOP-2 time periods using a nonparametric one-way analysis of variance approach (Kruskal-Wallis test). Dunn’s multiplecomparison test was used for pairwise posttest comparisons of values from each time period. P < .05 was considered statistically significant.
© American Society for Clinical Pathology
AJCP / ORIGINAL ARTICLE
Table 1 Changes in Standard Ordering Protocols Change No.
Category
SOP-1
Rationale
SOP-2
1
AML/MDS
Common SOP for AML and MDS
Some molecular and cytogenetic tests for AML not relevant in MDS Tests for MRD not necessary in follow-up marrows with overt disease
Separate MDS SOP created
2
3
4
Marrow failure
5
Lymphoma
6
MPN
7
8
Myeloma
For AML, all morphologically positive marrows categorized together with the same set of recommended tests Karyotype and MDS FISH panel performed at first presentation Karyotype and MDS FISH panel performed at first presentation Karyotype in overt disease and pretransplant marrows only Quantitative BCR-ABL testing in all follow-up marrows in CML JAK2 mutation testing in all follow-up marrows if previously positive Full myeloma FISH panel performed at pre-SCT evaluation
MDS FISH panel does not add unique information beyond karyotype MDS FISH panel does not add unique information beyond karyotype MDS-like karyotypes occasionally seen NCCN guidelines suggest following BCR-ABL in peripheral blood JAK2 mutation testing remains positive throughout follow-up in most cases FISH is almost always negative in morphologically negative follow-up marrows
Separate categories for persistent or relapsed disease (vs first diagnosis) with more focused test panel Only karyotype recommended at first presentation Only karyotype recommended at first presentation
Karyotype for all marrows No BCR-ABL molecular testing in follow-up marrows if peripheral blood is positive JAK2 mutation testing at diagnosis and pre- or posttransplant only No FISH at pre-SCT evaluation if the marrow is morphologically negative for disease
AML, acute myeloid leukemia; CML, chronic myeloid leukemia; FISH, fluorescence in situ hybridization; MDS, myelodysplastic syndrome; MPN, myeloproliferative neoplasm; MRD, minimal residual disease; NCCN, National Comprehensive Cancer Network; SCT, stem cell transplant; SOP, standard ordering protocol; SOP-1, standard operating protocol 12 months after DMT implementation; SOP-2, standard operating protocol 12 months after SOP revision.
Results Table 1 details the changes by disease category between the original SOPs (SOP-1) and the revised SOPs (SOP-2) as well as the rationale for the change. Two of the eight changes (changes 1 and 2) were made to clarify the SOPs based on user experience. Another change (change 6) was made to more closely align the SOPs with published guidelines. The remaining guidelines were based on data collected during implementation of the DMT. For example, our group published data indicating that fluorescence in situ hybridization (FISH) for common abnormalities observed in myelodysplastic syndrome (MDS) does not usually add unique information beyond that provided by routine karyotype in myeloid neoplasms.13 Consequently, MDS FISH testing was removed from the SOPs for acute myeloid leukemia (AML; change 3) and MDS (change 4). Similarly, data from the DMT showed that FISH on unsorted marrow aspirates (ie, no plasma cell selection) for
© American Society for Clinical Pathology 587
common abnormalities seen in plasma cell myeloma is usually negative in marrows without overt morphologic evidence of disease.7 Thus, FISH testing was eliminated in morphologically negative follow-up bone marrows in patients with myeloma (change 8). Most of these changes involved a reduction in the number of recommended tests. However, in one case (change 5), testing that had been eliminated in the SOP-1 was added back, based on data showing significant positive results for tests considered unnecessary by the initial DMT.7 The SOPs were revised 12 months after initial implementation of the DMT process. To determine the impact of these revisions on test ordering and results, we compared these data for 12 months after the revision (SOP-2) with those of the initial 12 months (SOP-1) and data collected for 6 months prior to implementation of the DMT (pre-DMT). Table 2 lists the characteristics of the bone marrow testing for each of these periods. There was statistically significant variation in the distribution of disease categories between the time periods. The fraction of cases in which the DMT approach of reflex testing
Am J Clin Pathol 2016;146:585-593 587 DOI: 10.1093/ajcp/aqw180
Seegmiller et al / ITERATION
OF
BONE MARROW TESTING PROTOCOLS
Table 2 Case Characteristics Characteristic
Pre-DMT SOP-1
SOP-2
Time, mo Bone marrows, No. Total No. of tests Disease groups, No. (%) ALL AML/MDS Marrow failure Lymphoma MPN Myeloma Other SOP utilization, No. (%) Reflex Nonreflex
6 780 2,919
12 1,806 6,202
12 1,926 5,708
38 (5) 241 (31) 63 (8) 134 (17) 45 (6) 222 (28) 37 (5)
82 (4) 719 (40) 105 (6) 267 (15) 87 (5) 462 (25) 84 (5)
82 (4) 636 (33) 148 (8) 355 (18) 146 (8) 439 (23) 120 (6)
P Value
