Integration of heterogeneous clinical decision support ...

Integration of heterogeneous clinical decision support systems and their knowledge sets: feasibility study with drug-drug interaction alerts Hye Jin Kam, PhD1, Jeong Ah Kim, PhD2, InSook Cho, PhD3, Yoon Kim, MD, PhD4, Rae Woong Park, MD, PhD5 1 Samsung Advanced Institute of Technology, Samsung Electronics; 2 Department of Computer Education, Kwandong University, Korea; 3Department of Nursing, Inha University, Korea; 4Department of Health Policy and Management, Seoul National University, Korea; 5Department of Biomedical Informatics, Ajou University School of Medicine, Korea Abstract There exist limitations in both commercial and in-house clinical decision support systems (CDSSs) and issues related to the integration of different knowledge sources and CDSSs. We chose Standard-based Shareable Active Guideline Environment (SAGE) as a new architecture with knowledge integration and a centralized knowledge base which includes authoring/management functions and independent CDSS, and applied it to Drug-Drug Interaction (DDI) CDSS. The aim of this study was to evaluate the feasibility of the newly integrated DDI alerting CDSS into a real world hospital information system involving construction of an integrated CDSS derived from two heterogeneous systems and their knowledge sets. The proposed CDSS was successfully implemented and compensated for the weaknesses of the old CDSS from knowledge integration and management, and its applicability in actual situations was verified. Although the DDI CDSS was constructed as an example case, the new CDS architecture might prove applicable to areas of CDSSs. Introduction Hospitals have adopted various types of clinical decision support systems (CDSSs) into their hospital information systems (HISs) for the purpose of reducing medical errors and to improve the quality of patient management and their safety1-4. In Korea, official DDI notifications are issued by the Minister of Health and Welfare, which are reviewed by the Korea Food & Drug Administration (KFDA). To prescribe an absolutely contraindicated drug-pair, a physician must submit an appropriate reason for the prescription, or reimbursement for will be refused, which is in accordance with this official regulation. Ajou University Medical Center (AUMC) is a tertiary teaching hospital, and accommodates 1,099 patient beds as well as other facilities, including 92 intensive care units and 19 operating rooms. There are about 3,500 outpatient visits per day. All medications for patients are prescribed using a Computerized Physician Order Entry (CPOE) system of EMR. All Korean hospitals are required to participate in this official program of governmental regulatory DDI knowledge of absolutely contraindicated drug pairs (about 380 rules) for DDI CDS service. AUMC had previously participated in governmental regulation for DDI CDS service in Korea. If any valid drug pairs of absolute contraindication (from the official DDI knowledge) are found in comparisons of the current and past medication orders within 24 hours, a pop-up alert would appear, and the physician could choose to cancel, change, or override the alert by providing a rationale for the pairing. Those official notifications are updated at irregular intervals. It is often difficult to revise the DDI Knowledge Base (KB) in time for individual institutions. Consequently, several commercial vendors provide an updating service for knowledge and the whole DDI CDSS; those of AUMC are also managed by commercial KB service and software5. However, it was a minimal rule set for regulatory purposes, which was limited to absolute contraindications and was not sufficient for clinically meaningful presentations of DDIs. Therefore, internal demands were raised for integrating and managing knowledge sets from different sources, such as therapeutic duplication, clinically significant adverse drug interactions, and clinical abuse/misuse according to internal guidelines that are not included in the governmental regulatory rules.

664

To respond to these issues, we have raised various CDS system requirements for optimal patient care4,6-8; those requirements for a multifunctional CDSS with non-redundant and non-volatile knowledge integration, a centralized KB which includes authoring/management functions, and an independent CDSS from HIS have extracted from literature reviews and discussions are summarized in Table 1. We also proposed an integrated CDSS involving construction of an integrated CDSS derived from two heterogeneous systems and their knowledge sets: [1] an old governmental regulatory CDS service with a commercial KB service, and [2] a newly adopted CDSS that provides a wide range of rules and processes based on standard terminology, which includes integrated knowledge, knowledge encoding, and data interface. The aim of this study was to evaluate the feasibility of the newly integrated DDI alerting CDS system into a real world HIS involving construction of an integrated CDS service derived from two heterogeneous systems and their knowledge sets. Table 1. A summary of requirements for a DDI CDSS. CDSS component

Requirements Integration from various resources should be possible4,6.

Knowledge

A non-redundant knowledge set should be prepared. Both regulatory and custom rules should not be erased or altered by the other’s updates7. An up-to-date knowledge management plan and process are necessary7. A centralized common knowledge repository is required4.

Knowledge Repository

Knowledge should be represented and managed in standard forms for portability and interoperability4,6,8.

Executing Engine Accuracy and timeliness are the first priority8. CDSS Process

A CDS process can be functionally independent from the central structure of HIS for portability9.

We chose Standard-based Shareable Active Guideline Environment (SAGE)10-12 as a new CDSS architecture that compensates for the requirements mentioned in Table 1, and it was applied to the clinical system of the AUMC in Korea. SAGE is an example of a service-based architectural approach, which is the fourth level of the four-phase architecture model of decision support by Wright and Sittig13. It is a knowledge representation tool with a JAVA-based stand-alone open architecture, which provides knowledge authoring functions to the clinician10,14. The SAGE workbench provides authoring tools employing Protégé11 and standard exports such as XML. We preferentially implemented the new CDS service into the DDI CDSS of AUMC, which urgently required an integration of knowledge bases.

Materials and Methods Clinically Required Additional Knowledge on DDI A new set of DDI knowledge encompassing all clinically significant DDIs was introduced to compensate for the KFDA national mandatory DDIs, which have a regulatory purpose, in addition to the absolute contraindication of governmental DDI rules. The additional clinically required DDI knowledge set with broader clinical range was developed collaboratively by research teams from eight universities and hospitals in Korea15; the new knowledge set was extracted from KFDA allowances, five databases (Micromedex, Drug Interaction Facts, Lexi-Comp online drug interaction, the Top 100 Drug Interactions and Clinical Phamacology online), eight nations’ allowances (USA, Japan, Germany, Canada, Switzerland, UK, Italy and France), and the opinions of expert associations (KMA and KPA). From the new DDI rule set, we selected contraindication (n=101) and major DDIs (n=608) by selecting DDIs that actually occurred in AUMC during the most recent 3 months and newly introduced to the new CDSS. We utilized these as an example set of newly introduced knowledge; other information from various resources or institution-specific knowledge can be used in an identical fashion.

665

Integration of the New CDSS with the AUMC Health Information System SAGE is a service-based architecture approach to CDSS implantation16-17. SAGE places a standardized API named Virtual Medical Record (VMR) at the side of a clinical system and enables the translation of a target hospital’s medical records into SAGE standard guideline formats13 based on the HL7 RIM. In this study, we utilized a SAGE-based CDS service with a publically available execution engine, named ‘U-brain’18. The engine and knowledge repository of the U-brain had been developed as a national project with the objective of inter-institutional portability and interoperability18-20. Knowledge Integration and Encoding into the New Repository The two knowledge sets had different origins, and were out-of-sync in management. Furthermore, as governmental regulatory DDI knowledge is updated irregularly, an automatic merging process needed to be designed. To perform an integrated DDI CDSS, an exclusive non-redundant and non-volatile knowledge set had to be prepared. An appropriate methodology was required for the integration and management of the two different rule sets (governmental and clinical requirements) without accidental exclusion during integration. Therefore, we described some of the issues associated with updates, integration, and duplicate removal. We proposed an integrated KB, which was automatically reconstructed daily into the integrated system via merging governmental regulatory DDIs on contraindication and the newly introduced one on absolute contraindication and major interaction, with the governmental DDI knowledge as a center (Figure 1). Information on governmental regulatory DDIs was maintained via internal and commercial drug codes. For the purpose of interoperability, we have selected the WHO Anatomical Therapeutic Chemical (ATC) codes as the standard terminology, and mapped both sides. To remove overlaps, we additionally integrated the two knowledge sets on the basis of governmental regulatory DDI knowledge, and stored them in the new repository.

Figure 1. Integrated DDI knowledge with daily reconstruction.

Data Interfaces As shown in Figure 2, the integrated CDSS was operated as a stand-alone web server outside of HIS, while the CPOE was in the HIS. An XML-based interface is defined between the two systems to obtain patients’ prescription data without losing the interoperability of the new CDSS. When a prescription order occurred, the list of ingredients, ATC codes, formula (injection, external or oral), internal drug codes, and prescription dates were transmitted to the new CDSS engine from current and previous orders (up to 24 hours) for each patient. On the engine side, a DDI list and related drug pairs among the received list were sent back to the CPOE with the following information: [1] alert type (governmental regulatory absolute

666

contraindication, newly imported absolute contraindication, or major interaction), [2] an identification code for additional detailed information sources for DDIs, [3] ATC codes, and [4] formula. The interface contents included both the standard ATC and internal drug codes, which connected the CPOE with the new CDSS. Demonstration The subject knowledge contents and department were limited to DDIs and the emergency department (ED) of AUMC. The new system was applied to the ED over a 3-month period by gradually increasing the DDI rules in a stepwise manner: The new CDSS was used for ED patients, whereas the old system was used for all other medication orders for inpatients and outpatients. ED was selected for the feasibility test of the new CDSS due to the following reasons: [1] prescriptions are urgently filled, [2] joint treatments are performed among several departments, [3] it has the characteristics of both hospitalization and outpatient protocols, and [4] greater concerns about medical errors are present owing to lack of patient information. Using this architecture, five sequential (3 weeks for each) pilot tests were conducted, as shown in Table 2. At stage 1, governmental regulatory knowledge and old CDS engine were applied. At stage 2, only the old CDS engine was replaced with the integrated one: the DDI CDS process, knowledge, and the contents and number of alerts were also identical to stage 1, assuming the same order situation. From stages 2 to 4, the amount of knowledge increased gradually in the order of governmental regulatory contraindications only, plus additional required contraindications, plus required major interactions. At stage 5, all of the settings were the same as in stage 4; additionally, the alerts due to the new knowledge were not provided to the clinicians while logging them to see the occurrence and pattern of DDI alerts.

Table 2. Five Sequential Demonstrations for the Integrated DDI CDSS. Number of applied rules* Absolute contraindications Additionally Stage Period Newly required Old Governmental Additionally introduced major regulatory required interactions 1 √ Nov 19, 2009 - Dec 09. 2009 360 2 √ Dec 10, 2009 - Dec 30, 2009 387 3 √ Dec 31, 2009 - Jan 20, 2010 389 101(22) 4 √ Jan 21, 2010 - Feb, 10, 2010 404 101(22) 608(592) 5 √ 101(22)† 608(592)† Feb 11, 2010 -Mar 03, 2010 404 * The numbers of rules in each knowledgebase are an average number of rules during each test period: ( ) = number of rules excluding duplicates with the governmental regulatory DDI rules † Those DDI rules were not provided to the clinicians while logging them CDSS

Results We implemented an integrated DDI alerting and auditing system, as shown in Figure 2. Components of commercial, in-house (original) and newly adopted systems are shown in dark gray, gray, and white, respectively. The non-functioning parts are shown in dotted lines. Initially, the two CDSSs were physically and logically separated from one another (Figure 3A). In Figure 3B, all of the contents and functions for the executing engine and the governmental regulatory DDI rules of HIS were transferred to the integrated CDSS to perform a stand-alone CDS function and formation of functional architecture, as shown in Figure 3C, by changing the pathway of the DDI searching process from the old system to the new one.

667

Figure 2. Simplified architectures of DDI CDSSs: (A) Two separated CDSSs- the old CDSS and newly adopted U-brain CDSS, (B) the integrated architecture of the CDSSs and (C) functional architecture of integrated CDSS. The proposed system derived from (A) to (B), and aimed to achieve (C) as the final architecture.

Evaluation on the Liability of the New Executing Engine A CDSS should obey the KB rules within the appropriate response time; this relies primarily on the performance of the executing engine used to assess DDI violations. We have performed a runtime performance test on whether the new system could get the responses within three seconds by generating an expected maximal load to the system. Through status analysis for the operating system, the number of simultaneous users was 150 at its peak-time, which was 10% of all users. When the average transaction time (3 seconds) and transaction think-time (20 seconds) were applied, the number of virtual users was 20. The runtime performance test was performed with 30 virtual users, the optimal number, and it showed that every transaction was obtained within three seconds. For system performance, we evaluated the decision-making accuracy of the system using the new CDSS based on randomly sampled historical prescription data, determining whether the new one displayed the same alerts when processing with the same patients’ prescription data and knowledge (mandatory contraindication rules=935). Among 4,781 prescriptions from 499 randomly selected patients, the new system reported 224 alerts, which corresponded completely to the alerts of the pre-existing system with 100% sensitivity and specificity. A Secondary, Fail-safe Process As mentioned before, a CPOE is the ‘mission critical’ function of HISs; the stability of a CDSS should be guaranteed. Considering the web-linked and separated structure of the new CDSS, we designed a ‘three seconds’ fail-safe process. If the new CDSS failed to give search results for drug combinations, the old system was run for the requests, in turn. Figure 3 shows the distribution of alerts by two systems from stages 1 to 5. Alerts of and by the fail-safe (old) system, during stages 2 to 5 accounted for 13.5%, 2.7%,

668

16.5%, and 0.3% of the total alerts during each stage, respectively. Considering the gradual increase of knowledge and the similarity of stages 4 and 5, returns to the old system were not attributed to the amount of knowledge. Operation of the fail-safe process resulted principally from failures of daily executions and re-operation of engine and database at the new CDSS server by other scheduled jobs and under other preestablished environments. The new CDSS was stabilized on February 13, 2009, during stage 5, and no fail-safe processes were detected after that time.

Figure 3. The distribution of alerts by two systems from stage 1 to 5. Brief Report for DDI Alerts and Prescriptions To evaluate the relevant functions via knowledge and the knowledge engine of the integrated CDSS, we conducted a pilot project by gradually increasing the DDI rules in a stepwise manner, as shown in Table 2. To estimate the scale of alert occurrence in the ED, we compared the number of prescriptions and prescribed patients during stage 1 (with the same applied DDI knowledge). The rate of alert occurrence in the ED was about 4.2-fold higher than that of the entire hospital: the number of prescriptions were 27,585 (for the entire hospital) and 2,496 (for the ED) cases per day, and the number of prescribed patients were 8,153 (for the entire hospital) and 963 (for the ED) cases per day. During the test period, the total number of prescriptions in the ED was 256,313 (daily mean=2,418). In addition, the rates of alerts were 0.57%, 0.55%, 0.54%, 1.19%, and 1.36%, respective stage numbers. Although the amount of overridden alerts also varied broadly according to stage, the percentages of alert overrides for each stage were 98.0%, 96.0%, 96.9% and 98.1%, respectively, as shown in Figure 4. We had anticipated an upward trend in alert overrides according to increases in the amount and the mildness of knowledge importance, as demonstrated in several reports6,10,16; this was hard to identify because there had already been high override rates at earlier stages with smaller amounts of knowledge. More detailed results of DDI alerts for this application have been reported by Kam et al21.

669

Figure 4. The number of alert overrides for each stage. Dotted part in stage 4 represents new alerts that were not provided to the clinicians: they were excluded from the calculation of percentages of alert overrides at stage 4. Applicability Verification of the Integrated DDI CDSS We proposed a new CDSS integrating the additionally required DDI rules and process of a new CDSS with the old DDI CDS system, involving construction of an integrated CDSS derived from two heterogeneous systems and their knowledge sets: [1] an old governmental regulatory CDS service with a commercial KB service, and [2] a newly adopted SAGE-based CDSS that provides a wide range of rules and processes based on standard terminology, as shown in Figure 5, which included integrated knowledge, knowledge encoding, and data interface. The proposed CDSS was successfully implemented to AUMC, and compensated for the weaknesses of the old CDSS from knowledge integration, management, and the ability to verify its applicability in actual situations.

Figure 5. Overall system architecture of the integrated DDI CDSS.

670

Discussion We corroborated the feasibility of an integrated CDSS via the combination of a regulatory and a clinically required rule set into an integrated CDSS based on standard terminology. The proposed CDSS was successfully implemented to AUMC, and compensated for the weaknesses of the old CDSS from knowledge integration, management, and the ability to verify its applicability in actual situations. Additionally, the DDI CDS service is currently expanded and applied to prescription orders of all inpatients and ICU patients. In many in-house CDSSs, the CDS processes depend heavily on the processes of a particular HIS and vary with the differing internal requirements of the institutes. However, a CDSS in the form of plug-in software, which is clearly separate from the central structure of HIS and linked to the CPOE from the outside, can protect the portability of the whole CDSS. We have implemented a system-independent and centralized common knowledge repository7 and utilized an XML interface along with a new executing engine. Other CPOE systems of HISs can be linked to this CDSS through the appointed interface design and contents, and be served the same DDI alerting service, regardless of their innate system structures. In this study, we used ATC codes for integration of DDI knowledge; if necessary, other coding schemes could be used, requiring only that the contents of rules and data be modified to render them capable of interface with the same system structure. Kuperman et al. previously referenced the importance of preservation of customized and compulsory knowledge from updates of vendor knowledge as another constraint for KB customization6. We have preserved both of the knowledge sets and simultaneously maintained the changing governmental regulatory DDIs by using two KBs in conjunction to pivot around the regulatory DDIs. We also realized that the need for alert loggings was profound with regard to research into DDI alerts and associated knowledge management. A tailored logging is distant from the current off-the-shelf CDSSs7. More detailed data on occurrence and patterns of DDI alerts and the clinicians’ response to the alerts could be obtained by sophisticated auditing functions; this information will enable implementation of a more effective and efficient CDS service. We are conducting a randomized clinical trial (RCT) on DDI alerts to evaluate the clinical effectiveness of those alerts for patient safety. We have transferred the functions of pre-existing CDSS to the new general-purpose system with the open architecture of the SAGE workbench and the knowledge authoring/browsing tool of Protégé11; this ensures secure system flexibility and allows for user-centered knowledge management. The CDSS proposed herein could express guideline and activity graphs, math functions, and restrictions (not shown in this paper); additionally, a host of other useful applications are available in U-brain, including clinical guidelines.25 Conclusion In this study, we proposed an architecture that integrates the process of a new SAGE-based CDSS and the old DDI CDSS to satisfy the various and continuous inner/outer requests on rules and CDSSs. The proposed CDSS was successfully implemented to AUMC, and compensated for the weaknesses of the old CDSS from knowledge integration, management, and the ability to verify its applicability in actual situations. Although the integrated DDI CDSS was constructed as an example case, the new CDS architecture might prove applicable to areas of CDSSs other than the DDI.

Acknowledgement This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MEST) (No. 2010-0023402 and 2011-0018258). All of the authors of this work hereby declare that there were no conflicts of interest or any other relationships that could have inappropriately influenced this study.

671

References 1.

2. 3.

4.

5. 6.

7.

8.

9. 10. 11. 12. 13.

14. 15. 16. 17. 18.

19. 20.

21.

Zwart-van Rijkom JE, Uijtendaal EV, ten Berg MJ, van Solinge WW, Egberts AC. Frequency and nature of drug-drug interactions in a Dutch university hospital. Br J Clin Pharmacol. 2009;68(2):18793. Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med. 2003;163(21):2625-31. Hsieh TC, Kuperman GJ, Jaggi T, Hojnowski-Diaz P, Fiskio J, Williams DH, et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc. 2004;11(6):482-91. Kuperman GJ, Bobb A, Payne TH, Avery AJ, Gandhi TK, Burns G, et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc. 2007;14(1):29-40. Kuperman GJ, Reichley RM, Bailey TC. Using commercial knowledge bases for clinical decision support: opportunities, hurdles, and recommendations. J Am Med Inform Assoc. 2006;13(4):369-71. Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10(6):523-30. Sittig DF, Wright A, Simonaitis L, Carpenter JD, Allen GO, Doebbeling BN, et al. The state of the art in clinical knowledge management: an inventory of tools and techniques. Int J Med Inform. 2010;79(1):44-57. Borbolla D, Otero C, Lobach DF, Kawamoto K, Saldano AMG, Staccia G, et al. Implementation of a clinical decision supprot system using a service model: results of a feasibility study. Medinfo. 2010;2010:816-20. SAGE Guideline Model. Available at: http://sage.wherever.org/model/model.html. [cited 2011 March 11]. Protégé as a Guideline Modeling Tool. Available at: http://sage.wherever.org/model/modeltools.html. [cited 2011 March 11]. Greenes RA. Clinical decision support : the road ahead. Califonia, Elsevier;2007. Wright A, Sittig DF. A four-phase model of the evoluation of clinical decision support architectures. Int J Med Inform. 2008;77(10):641-9. Kim JA, Cho I, Kim Y. Knowledge translation of SAGE-based guidelines for executing with knowledge engine. Proceedings of the AMIA Annual Symposium; 2008 Nov 8-12; Washington DC, USA. KIMS-Online. Seoul: UBM Medica Korea Ltd.;c1998-2010 Available at: http://new.kimsonline.co.kr. [cited 2011 March 11]. Lee S, Information System of National Drug Standards and Knowledgebase. Health and Medical Technololgy R&D Program , Project no. D060002, Ministry of Health and Welfare, Korea, 2006. Parker GG, Rocha RA, Campbell JR, Tu SW, Huff SM. Detailed clinical models for sharable, excutable guidelilnes. Medinfo. 2004;2004:145-8. Ram P, Berg D, Tu S, Mansfield G, Ye Q, Abarbanel R, et al. Executing clinical practice guidelines using the SAGE excution engine. Medinfo. 2004;2004:251-5. Kim JA, BinGuShim, SunTaeKim, JaeHoonLee, InSookCho, Kim Y, editors. Translation protege knowledge for execution. Proceedings of 11th International Protege Conference; 2009 Jun 23-26; Amsterdam, Netherlands. Kim JA, Cho I, Kim Y. Clinical decision support service architecture for EHR perspectives in Korea. Proceedings of the AMIA Annual Symposium; 2008 Nov 8-12; Washington DC, USA. Kim JA, Cho I, Kim Y. Knowledge translation of SAGE-based guidelines for executing with knowledge engine. Proceedings of the AMIA Annual Symposium; 2008 November 8-12; Washington DC, USA. p. 1008. Kam HJ, Park MJ, Kim WJ, Yoon DY, Ahn EK, Park RW. Knowledge integration and use-case analysis for a customized drug-drug interaction CDS service. In: Kim T-h, Stoica A, Chang R-s, editors. Communications in Computer and Information Science, Proceedings of the 1st International Conference on Security-enriched Urban Computing and Smart Grid; 2010 September 15-17; Daejeon, Korea: Springer; 2010.

672

22. Mille F, Schwartz C, Brion F, Fontan JE, Bourdon O, Degoulet P, et al. Analysis of overridden alerts in a drug-drug interaction detection system. Int J Qual Health Care. 2008;20(6):400-5. 23. Trivedi MH, Daly EJ, Kern JK, Grannemann BD, Sunderajan P, Claassen CA. Barriers to implementation of a computerized decision support system for depression: an observational report on lessons learned in "real world" clinical settings. BMC Med Inform Decis Mak. 2009;9:6. 24. Lin CP, Payne TH, Nichol WP, Hoey PJ, Anderson CL, Gennari JH. Evaluating clinical decision support systems: monitoring CPOE order check override rates in the Department of Veterans Affairs' Computerized Patient Record System. J Am Med Inform Assoc. 2008;15(5):620-6. 25. Kim J, Cho I, Lee J, Shim D, Kim H, Kim Y. Development and Verification of HT encoding model. Proc AMIA Annu Symp; 2008 November 8-12; Washington DC, USA.

673