European Journal of Cardio-thoracic Surgery 28 (2005) 400–406 www.elsevier.com/locate/ejcts
An attempt at data verification in the EACTS Congenital Database* Bohdan Maruszewskia,*, Francois Lacour-Gayetb, James L. Monroc, Bruce E. Keoghd, Zdzislaw Tobotaa, Andrzej Kansya b
a The Children’s Memorial Health Institute, Warsaw, Poland Denver Children’s Hospital, University of Colorado, Denver, CO, USA c Southampton General Hospital, Southampton, UK d The Heart Hospital, Westmoreland Street, London, UK
Received 24 November 2004; received in revised form 15 February 2005; accepted 30 March 2005
Abstract Objective: The multi-national and multi-institutional collection of data on outcomes in Congenital Heart Surgery (CHS) provides the possibility of analysis of results of treatment and may result in further improvement of the quality of care. The verification of data as far as the completeness and accuracy is necessary to give confidence to all sides—the patients, centers and regulatory authorities. The source data verification (SDV) although difficult, appears possible even in such a large-scale database. Methods: Out of 5.274 patients and 5.612 procedures data of 2003, collected in the database, 1.703 (32.3%) and 1.895 (33.8%), respectively, have been verified at five sites on following fields: IPPV time, date of birth, date of admission, date of surgery, date of discharge/mortality, body weight, case category, CPB time, AoX time, Circulatory arrest time. SDV was performed at five sites by two database officers using the sources of information different to the local copy of the database (patients’ files, operation notes, perfusion charts, OR Books). Verification was performed between June 1st and July 31st 2004. Statistical analysis was performed using R-project software, ver. 2.0.0. and Welch’s t-test for comparison of continuous variables. P-value O0.05 was used as statistically significant difference between groups. Results: Pre- and post-verification mortalities in all groups showed no significant differences although seven deaths out of 68 (10.27%) were missed. None of the other verified fields showed significant differences after verification. Conclusions: Source Data Verification showed no statistically significant differences between verified and non-verified data on 30 days mortality, LOS, age, body weight, CPB time, AoX and Circulatory arrest time. IPPV time was not available in 58.6% procedures. Q 2005 Elsevier B.V. All rights reserved. Keywords: Congenital heart surgery; Quality of care; Outcome analysis; Database; Data verification; Data validation
1. Introduction The continuous evaluation of the quality of care became one of the highest priorities in modern surgical practice [1]. Especially, in the field of Congenital Heart Surgery (CHS), where adverse outcomes can be frequent due to the wide spectrum of congenital cardiac conditions and severity of pathologies, as well as high proportion of palliations, it is important to know the exact information on mortalities and morbidities [2,3]. Even more important is the possibility of the comparison of outcomes between surgeons and institutions on large, international, multi-institutional scale [4–8]. In this spirit in early 1990s, a group of congenital; heart surgeons of the European Congenital Heart Surgeons
* Presented at the joint 18th Annual Meeting of the European Association for Cardio-thoracic Surgery and the 12th Annual Meeting of the European Society of Thoracic Surgeons, Leipzig, Germany, September 12–15, 2004. * Corresponding author. Tel.: C48 22 815 73 46; fax: C48 22 815 73 50. E-mail address:
[email protected] (B. Maruszewski).
1010-7940/$ - see front matter Q 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.ejcts.2005.03.051
Foundation (Association since 2003, ECHSA) decided to create the European Congenital Heart Defects Database [9]. This project was supervised by M. Elliott and based at the Great Ormond Street Hospital for Sick Children, London, UK. The data on 16.000 procedures had been collected until 1998, when this database was moved to Warsaw, Poland under responsibility of B. Maruszewski of the Children’s Memorial Health Institute [9,10]. Since 2000, the European Association for Cardio-Thoracic Surgery (EACTS) has accepted this project and took substantially the financial responsibility for its further development together with ECHSA. At the same time, the simultaneous efforts were made at the Society of Thoracic Surgeons (STS) where the STS National Congenital Heart Surgery Database had been developed under the leadership of C. Mavroudis of the Children’s Memorial Hospital, Chicago, USA [4–7,11,12]. These initial efforts provided the impetus for further cooperation between Congenital Heart Surgeons from both continents and initiated the International Congenital Heart Surgery Nomenclature and Database Project [11,12,13,15,16]. Its major aims were to establish a common International Nomenclature for CHS and the Minimum Data Set that have been used and updated continuously since year 2000 in both databases [11,13].
B. Maruszewski et al. / European Journal of Cardio-thoracic Surgery 28 (2005) 400–406
401
Table 1 Standard EACTS Congenital Database online report of 18th of September 2004
IPPV (h) Total CPB time (min) Total AorticX time (min) Circulatory arrest (min) Weight (kg) Age at operation (months) LOS (days)
No. of cases
% of all
Min
Mean
SD
Max
13.146 16.725 15.121 1.844 22.583 23.684 23.339
55.39 100.00 90.41 11.03 95.15 99.79 98.33
1 2.00 1.00 1.00 0.50 0.00 0.00
55.21 103.89 56.55 30.61 14.36 52.08 18.95
165.63 71.61 40.43 22.75 17.23 102.50 26.11
300 days 999.00 998.00 153.00 128 932.06 895.00
IPPV, intermittent positive pressure ventilation (measured as intubation time), refers to all procedures; CPB, cardiopulmonary bypass; AorticX time, aortic crossclamp time, refers to CPB procedures; LOS, length of stay, surgery–discharge time; Circulatory arrest refers only to CPB procedures.
Until September 2004, the data on 23.735 CHS procedures have been collected, analyzed and reported online at the EACTS Congenital Database (www.eactscongenitaldb.org) (Table 1). The reliability of any data collection remains the key issue when the information is used for further outcome analysis and comparison. That is why data completeness and accuracy is essential, although it remains one of the major challenges for database management [4,12,14,15]. The meaning of ‘data validation’ should be understood as testing internal data consistency, e.g. the date of operation should not be earlier than the date of birth. Data validation is possible at data entry (surgeon, software) and in the database before data analysis. The EACTS Congenital Database developers have provided the internal database validation mechanism that automatically excludes non-accurate data according to agreed criteria (www.eactscongenitaldb.org). The next key step towards improvement of the data quality is the efficient data verification [17]. According to the Good Clinical Practice guidelines defined and recommended by World Health Organization (WHO), all database contents should be verified by comparison with the original records at the site of their origin—Source Data Verification (SDV) [17,18]. SDV is an evaluation of the conformity of database contents with the source data. SDV, although very difficult on international, multiinstitutional scale due to its costs, time consuming and lingual problems, appears to be possible and remains essential. For all reasons described above the EACTS Congenital Database management decided to make an attempt at data verification and check on the potential discrepancies between verified and non-verified data. 2. Material Five European CHS Centers volunteered to participate in the project and undergo 2003 data verification at their sites. The verification was conducted between 1st of June and 31st of July 2004 and concerned the following fields: † 30 days mortality † LOS † IPPV and additionally: † date of birth † date of admission
† † † † † † †
date of surgery date of discharge/mortality body weight Case category CPB time AoX time Circulatory arrest time
Out of 5.274 patients and 5.612 procedures data of the year 2003 collected in the database 1.703 (32.3%) and 1.895 (33.8%) have been verified, respectively (Table 2).
3. Methodology SDV has been conducted by EACTS Congenital Database staff (two officers). The data verification process was organized in the following steps: † 100% of patients data of 2003 have been supplied to the Database before site visit. † The data underwent standard internal validation procedure. † Person locally responsible for the verification indicated the sources of information at the site, different to the local copy of the database or related to the database which were as follows: – patients’ files – hospital computerized system – operation notes – perfusion charts – OR books † The database staff visited the site and checked on the accuracy of 100% of 2003 records for the fields undergoing verification. The central database printouts have been compared with the local sources. Table 2 Material Center
Patients
Procedures
A B C D E All
281 444 199 362 417 1703
305 480 227 385 498 1895
No. of patients and procedures by centers.
402
B. Maruszewski et al. / European Journal of Cardio-thoracic Surgery 28 (2005) 400–406
Table 3 EACTS Congenital Database verification data set
5. Results
Patient no 2; local ID: 00109/03 OPERATION_ID: XXX_2003-05-07_20:04:16 DATE_OF_BIRTH: 2001-05-09 DATE_OF_SURGERY: 2003-01-06 DATE_OF_ADMISSION: 2003-01-05 DATE_OF_DISCHARGE: 2003-01-17 WEIGHT: 12.3 CASECATEGORYa: CPB CPB_TIME: 61 AOX_TIME: 28 DATE_OF_MORTALITY: IPPV: 5 Is correct (#) Is not correct (!)
† The statistical analysis was performed on both sets of data (verified, non-verified) for comparison (Table 3).
Two patients, out of 1.705 after verification, have been missed. There have been no statistically significant differences between pre- and post-verification mortalities in all verified groups of patients, nevertheless seven deaths were missed out of 68 deaths found after verification (10.27%) (Tables 4–7). There have been no statistically significant differences found between verified and non-verified data in three groups of patients: all, infants and neonates, as far as age, AoX, Circulatory arrest, CPB time, LOS, body weight. Though only two patients were not reported out of 1.705 (after verification), there have been bidirectional shifts between age groups due to the correction in dates of birth and/or of operation (Table 8). IPPV time was not available in three Centers.
4. Statistical methodology
6. Conclusions
Results were expressed as mean values. All analyses were performed with appropriate software (R-project, ver. 2.0.0). Comparisons of continuous variables were performed by Welch’s t-test. P-value greater than 0.05 means that difference between groups is not statistically significant.
Source Data Verification showed no statistically significant differences between verified and non-verified data on 30 days mortality, LOS, age, body weight, CPB time, AoX and Circulatory arrest time IPPV time was not available in 58.6% procedures.
a
Case category: CPB, non-CPB, Thoracic, ECMO, Interventional cardiology.
Table 4 Verification results—30 days mortality Before verification
All Infants Neonates
After verification
P-value
No. of pts
No. of deaths
30 days mortality
No. of pts
No. of deaths
30 days mortality
1703 570 303
61 20 30
3.58% 3.51% 9.84%
1705 574 305
68 23 33
3.99% 4.01% 10.89%
0.534 0.658 0.670
Table 5 Verification results—all patients
Age (days) AoX time (min) Circulatory arrest time (min) CPB time (min) LOS (days) b.w. (kg)
Before verification, NZ1703
After verification, NZ1705
Mean
SD
Mean
SD
1559.594 55.266 37.53 128.233 15.276 15.256
3001.692 46.280 13.185 83.382 22.672 18.879
1555.378 56.413 36.98 130.799 15.782 15.164
2999.491 48.567 13.121 89.095 26.157 18.758
P-value
0.966 0.527 0.914 0.436 0.525 0.881
CPB, cardiopulmonary bypass; AoX time, aortic cross-clamp time; LOS, length of stay, surgery–discharge time; b.w., body weight.
Table 6 Verification results—infants
Age (days) AoX time (min) Circulatory arrest time (min) CPB time (min) LOS (days) b.w. (kg)
Before verification, NZ570
After verification, NZ574
Mean
SD
Mean
SD
158.528 57.236 27.15 124.851 18.211 5.610
88.884 38.643 10.883 63.149 27.618 4.008
158.204 58.032 26.50 127.026 19.181 5.444
88.594 39.957 10.788 65.659 30.675 2.132
CPB, cardiopulmonary bypass; AoX time, aortic cross-clamp time; LOS, length of stay, surgery–discharge time; b.w., body weight.
P-value
0.946 0.749 0.925 0.593 0.540 0.347
B. Maruszewski et al. / European Journal of Cardio-thoracic Surgery 28 (2005) 400–406
403
Table 7 Verification results—neonates
Age (days) AoX time (min) Circulatory arrest time (min) CPB time (min) LOS (days) b.w. (kg)
Before verification, NZ303
After verification, NZ305
P-value
Mean
SD
Mean
SD
11.003 75.519 17.406 173.808 22.452 3.008
7.433 45.787 24.801 93.754 28.846 1.284
10.969 76.197 17.425 178.423 23.882 3.033
7.422 46.940 24.878 99.066 31.239 1.302
0.954 0.889 0.994 0.954 0.546 0.954
CPB, cardiopulmonary bypass; AoX time, aortic cross-clamp time; LOS, length of stay, surgery–discharge time; b.w., body weight.
7. Discussion We believe that this study is the first one to present the verification process on the big, multi-institutional and international data registry of the outcomes of the CHS. It is obvious that 30 days mortality measure carries important limitation as far as the assessment of even early survival [8]. Undoubtedly, modern intensive care technology extends critical patients survival time what does not necessarily mean increase in long-term survival and improvement of the quality of life. Nevertheless this has been the only objective measure available 6 months after study time frames (2003). In future works, 30 days mortality measure should be avoided and probably survival analysis should concentrate more on mid-term outcome (6 months or 1 year). This is hard to believe that such a data can be obtained completely and accurately, especially in international big scale registry. Another limitation of the study is due to arbitrary selection of verified Centers. Relatively optimistic results of this first attempt at data verification, performed on the Centers being ‘good data submitters’, may not reflect the overall quality of data in the whole data collection. The only solution to decrease the negative impact of arbitrary choice of Centers will be using ‘by chance’ mechanism for the next 2005 attempt or include the Centers with relatively short experience in database works. The methodology employed in this project was very demanding, since 100% of 2003 records for nine fields have been checked, which finally verified seven investigated parameters. The verification process according Good Clinical Practice for the clinical trials, especially for the pharmaceutical products should cover 100% of data. This rule is very rarely applied to any verification on the big data collection, Table 8 IPPV time Center
No. of procedures
IPPV availability
A B C D E All
306 480 227 385 499 1897
306 480 No data No data No data 786 (41.4%)
IPPV, intermittent positive pressure ventilation (measured as intubation time).
although some institutions have used it to verify their clinical data (e.g. CHOP, Pennsylvania). There are important differences between various verification protocols used for medical data sets [15]. The verified samples vary from as small as 10 up to 100%. It shows that an International Committee of experts is needed to define common Data Verification methodology and to apply it in future works on outcome analysis in CHS. Such a group has been recently established between the EACTS and STS. This is the work in progress and some feedback from this Committee is expected before the end of 2005. Although this study is only an attempt at data verification within big, international, multi-centric database, it has proven the scientific value of such a data collection. The verification process will be continued and further works will focus on improvement of the quality of data. This should result in better understanding of the specific risk factors and areas of weakness in the quality of care in CHS.
Acknowledgements We would like to acknowledge doctors Duccio Di Carlo, Mark Hazekamp, Victor Hraska, Giovanni Stellin, and all staff members of five verified hospitals for the support and understanding during demanding process of data verification.
References [1] Daenen W, Lacour-Gayet F, Aberg T, Comas JV, Daebritz SH, DiDonato R, Hamilton JR, Lindberg H, Maruszewski B, Monro J. EACTS congenital heart disease committee. Optimal structure of a congenital heart surgery department in Europe. Eur J Cardiothorac Surg 2003;24(3): 343–51. [2] Williams WG, McCrindle BW. Practical experience with databases for congenital heart disease: a registry versus an academic database. Semin Thorac Cardiovasc Surg Pediatr Card Surg Annu 2002;5(1):132. [3] Mavroudis C (Chairman) and Congenital Database Subcommittee: Backer CL, Bove E, Burke RP, Cameron D, Drinkwater DC, Edwards FH, Grover FL, Hammon JW Jr, Jacobs JP, Kron IL, Mayer JE, Myers JL, Ring WS, Siewers RD, Szarnicki RJ, Watson DC Jr. Data Analyses of the Society of Thoracic Surgeons National Congenital Cardiac Surgery Database, 1994– 1997. Summit Medical, Minnetonka, MN; September 1998. [4] Mavroudis C, Gevitz M, Rings WS, McIntosh CL, Schwartz M. The society of thoracic surgeons national congenital heart surgery database: analysis of the first harvest. Ann Thorac Surg 1994-1997;68(2):601–24. [5] Jacobs JP, Mavroudis C, Jacobs ML, Lacour-Gayet FG, Tchervenkov CI, Gaynor JW, Clarke DR, Spray TL, Maruszewski B, Stellin G, Elliott MJ, Dokholyan RS, Peterson ED. Lessons learned from the data analysis of
404
[6]
[7]
[8]
[9]
[10]
[11]
[12] [13]
[14]
[15]
[16] [17] [18]
B. Maruszewski et al. / European Journal of Cardio-thoracic Surgery 28 (2005) 400–406 the second harvest (1998–2001) of the society of thoracic surgeons (STS) congenital heart surgery database. Eur J Cardiothorac Surg 2004;26(1): 18–37. Jacobs JP, Jacobs ML, Mavroudis C, Lacour-Gayet FG. Executive summary: the society of thoracic surgeons congenital heart surgery database—second harvest—(1998–2001) beta site test. The Society of Thoracic Surgeons (STS) and Duke Clinical Research Institute (DCRI), Duke University Medical Center, Durham, North Carolina, United States; Fall 2002 Harvest. Jacobs JP, Jacobs ML, Mavroudis C, Lacour-Gayet FG. Executive summary: the society of thoracic surgeons congenital heart surgery database—third harvest—(1998–2002). The Society of Thoracic Surgeons (STS) and Duke Clinical Research Institute (DCRI), Duke University Medical Center, Durham, North Carolina, United States; Spring 2003 Harvest. Stark J, Gallivan S, Lovegrove J, Hamilton JRL, Monro JL, Pollock JCS, Watterson KG. Mortality rates after surgery for congenital heart defects in children and surgeon’s performance. Lancet 2000;355(9208):948. Maruszewski B, Tobota Z. The European congenital heart defects surgery database experience: pediatric cardiothoracic surgical registry of the European association for cardio-thoracic surgery. Seminars in thoracic and cardiovascular surgery. Pediatr Cardiac Surg Annu 2002;5(1):143–7. Jacobs JP. Software development, nomenclature schemes and mapping strategies and for an international pediatric cardiac surgery database system. Seminars in yhoracic and cardiovascular surgery. Pediatr Cardiac Surg Annu 2002;5(1):153–62. Lacour-Gayet F, Maruszewski B, Mavroudis C, Jacobs JP, Elliot MJ. Presentation of the international nomenclature for congenital heart surgery. The long way from nomenclature to collection of validated data at the EACTS. Eur J Cardiothorac Surg 2000;18(128):35. Mavroudis C, Jacobs JP. Congenital heart disease outcome analysis: methodology and rationale. J Thorac Cardiovasc Surg 2002;123(1):6–7. Gaynor JW, Jacobs JP, Jacobs ML, Elliot MJ, Lacour-Gayet F, Tchervenkov CI, Maruszewski B, Mavroudis C. International congenital heart surgery nomenclature and database project. Congenital heart surgery committees of the society of thoracic surgeons. European association for cardio-thoracic surgery. Congenital heart surgery nomenclature and database project: update and proposed data harvest. Ann Thorac Surg 2002;73(3):101. Maruszewski B, Lacour-Gayet F, Elliott MJ, Gaynor JW, Jacobs JP, Jacobs ML, Tchervenkov CI, Kurosawa H, Mavroudis C. International congenital heart surgery nomenclature and database project. Congenital heart surgery committees of the society of thoracic surgeons, European association for cardio-thoracic surgery. Congenital heart surgery nomenclature and database project: update and proposed data harvest. Eur J Cardiothorac Surg 2002;21(1). Gibbs JL, Monro JL, Cunningham D, Rickards A. Survival after surgery or therapeutic catheterisation for congenital heart disease in children in the United Kingdom: analysis of the central cardiac audit database for 2000-1. Br Med J 2004;328:611. Mavroudis C, Jacobs JP, editors. Congenital heart surgery nomenclature and data base project. Ann Thorac Surg. 2000; 69: S1–372. Khosla R, Verma DD, Kapur A, Khosla S. Efficient source data verification. Indian Journal of Pharmacology 2000;32:180–6. WHO guidelines for Good Clinical Practice for Trials on Pharmaceutical Products; 1994.
Appendix A. Conference discussion Dr J. Monro (Southampton, United Kingdom): I do congratulate you on all this work and the remarkable database that you’ve collected for all these years. The results of the verification in these five units are amazing. With the British experience in mind, it is incredible that they only missed two patients and those seven deaths. I wonder if this is because you chose the five best, most organized centers, who therefore did have these very accurate data. When you’ve looked into the others, I think you may find that it’s not quite so good. But as you’ve said before, everything has got to be 100%, both centers and the data they’re collecting. Dr Maruszewski: And of course, you are right. We have taken the centers that offered us cooperation. And I have to use the opportunity to thank my colleagues from the five centers for their very hard work, because this is a lot of very hard work. In the future, we want to make the decision about the
centers that we choose by chance, so machine will choose the centers and then we will see. Mr V. Tsang (London, United Kingdom): The work you have done is very admirable, but can I talk about how to tackle a big problem in general terms? For example, your database is huge. And in psychological terms, when you have a huge problem people feel overwhelmed as to what to do next. I think the way to solve that sort of problem is to break the big problem into smaller blocks. The progress made in each of the smaller blocks, when added together, would become major progress. Dr Maruszewski: Those of you who saw the outcome analysis that we were showing on various occasions, including yesterday at the business meeting, probably understand that is exactly what we are trying, we are attempting to do. So take part of the data and clear them and make sure they are very fine, and then to conclude on one question that you want to answer. That’s what we are doing with neonates, Norwoods, influencing the overall outcomes. Dr B. Williams (Toronto, Ontario, Canada): I have tremendous respect for what you’ve done with this database and the enormous amount of work that you put into it. I’d like to comment on your findings that the time to extubation was much more difficult to find. That doesn’t surprise me because it’s data that’s out of our domain. It’s kept by other people in other areas. And my bias would be that we should not try to capture that information in a surgical database. Surely the intensivists have their own database and would it not be far more logical to create a link between your surgical registry and the database in the other areas of the hospital, specifically the intensive care unit. If you carry this analogy one step further and look at the genetics, I think our data about genetics is going to be extremely poor and we will need to link to the genetics department to find out which patients have genetic disorders. Dr Maruszewski: As you know, we are meeting beginning of October, CHSS and our association, in Montreal. One of the tasks is to make sure that both associations realize that IPPV time is a weak measure. And I think we should not carry on promoting this as the quality of care measure, because, first of all, we don’t have this measure, we don’t have the data, I agree. Dr G. Ziemer (Tuebingen, Germany): Regarding ICU-data we have to keep in mind that ICU care is organized so differently across Europe and even within certain countries, that uniform data collection is not practical at the moment, although the idea I think is a great one. Dr W. Mrowczynski (Poznan, Poland): Although the database is voluntary, the data verification may require agreement of the authorities governing, for example, cardiac surgery department. Don’t you think it can be an obstacle in data verification? Dr Maruszewski: It hasn’t been for those who keep sending data for years; and this problem has been sorted out at the European union level and the same in States. The parents of the patient who is admitted to the hospital, on the admission accept that the data are used for the medical record and for the medical database. It may be different in various states and, for example, I’m not aware of that in details, but so far we are able, in the legal manner, to collect this data. Dr F. Lacour-Gayet (Denver, Colorado, USA): I would like to congratulate you and your group in Warsaw for the incredible amount of work that produced for the last 10 years in finally building what is probably the best congenital heart surgery database in the world. Let me give you some information from the STS database. We have also decided to verify data at the STS. It’s a bit more complicated and there are some authorizations that are needed, but we are proceeding. We have decided on a protocol that is based essentially on comparing the data of the hospital registry to those of the department and check that there is no death missing. Ultimately, there is a need for a common protocol to verify data. Dr Maruszewski: Before I finish, I would like to say just a few words. I would like to give my best thanks and a lot of respect to our Association for the support that has always been given to the database even in a very premature state when the outcome was not sure. Then I would like to thank Bill Williams for his stimulation and motivation and criticism that we received some time ago and that motivated us to work. I would like to thank all the colleagues in the European countries for putting the data in and the effort made on verification. At the end, I would like to thank Dr Tobota from Warsaw who is running, in fact, this database. He is doing 90% of the work and I am just presenting. So thank you, Dr Tobota, very much.
