Pharmacovigilance
Evaluation of the Quality of Publications on Randomized Clinical Trials Using the Consolidated Standards of Reporting Trials (CONSORT) Statement Guidelines in a Spanish Tertiary Hospital Francisco Dasí, PhD, María Mercedes Navarro-García, BSc, Mercedes Jiménez-Heredia, PhD, Jose Magraner, MD, PhD, Juan R. Viña, MD, PhD, Federico V. Pallardó, MD, PhD, Andres Cervantes, MD, PhD, and Esteban Morcillo, MD, PhD
The main reason for conducting a clinical trial (CT) is to test the effect of a drug or medical procedure to improve treatment of a disease. CTs contribute most when they are rigorously conducted and the results are published adequately. The aim of this study is to assess, using the CONSORT statement guidelines, the quality of reporting of completed CTs conducted at a tertiary hospital to determine which sections of the articles should be improved. CTs published between 2002 and 2008 were identified by searching the MEDLINE and Cochrane Library. Forty of 127 completed CTs were published. There was a marked increase in the number of articles and the quality of the journals that published the CTs over time. Although the articles were published in high-impact index journals, the Consolidated
Standards of Reporting Trials (CONSORT) score reporting quality of the articles varied substantially, which indicates that they should be improved. The title, abstract, introduction, and discussion sections received the highest CONSORT scores and need little improvement. Poor reporting of methodological details and discussion on limitations and strengths were observed. In conclusion, much improvement remains to be made in the quality of reporting of CTs to allow reliable quality assessment of published trials.
C
misinterpretation of results due to inappropriate reporting of well-conducted CTs can lead to significant biases and, therefore, misleading conclusions that could affect the national health system at all levels, from therapeutic decision making at the individual level to changes in national health policies. The correct evaluation of a CT is only possible if the design, execution, and analysis of the results are properly described in scientific articles reviewed by experts in the field. Ideally, the article should contain detailed information on the design, methodology, and analysis. Moreover, the results should be generalizable so that the information will help us to understand the internal and external validity of a CT and will ultimately redound to the benefit of the patients. However, much evidence suggests that the quality of publications on the CT is far from optimal. A series of studies that began in 1995 and extend to the
linical trials (CTs) provide, if well designed and implemented, the best evidence on the effectiveness and interventions in the health field. However,
From Fundación Investigación Hospital Clínico Universitario de Valencia/INCLIVA, Valencia, Spain (Dr Dasí, Dr Navarro-García, Dr Viña, Dr Cervantes, Dr Morcillo); School of Medicine, Department of Physiology (Dr Dasí, Dr Pallardó), Department of Pharmacology (Dr Morcillo), and Department of Biochemistry (Dr Viña), University of Valencia, Valencia, Spain; Pharmacy Unit (Dr Jiménez-Heredia, Dr Magraner) and Oncology/ Hematology Unit (Dr Cervantes), Hospital Clínico Universitario de Valencia, Valencia, Spain; and Centro de Investigación Biomédica en Red de Enfermedades Raras (CIBERER), Valencia, Spain (Dr Pallardó). Submitted for publication December 13, 2010; revised version accepted March 29, 2011. Address for correspondence: Dr Francisco Dasí, Fundación Investigación Hospital Clínico Universitario de Valencia/ INCLIVA, Avda. Blasco Ibáñez, 17, E-46010 Valencia, Spain; e-mail:
[email protected]. DOI: 10.1177/0091270011407916
1106 • J Clin Pharmacol 2012;52:1106-1114
Keywords: CONSORT; randomized clinical trials; quality; reporting Journal of Clinical Pharmacology, 2012;52:1106-1114 © 2012 The Author(s)
EVALUATING THE QUALITY OF PUBLICATIONS USING CONSORT GUIDELINES present lead to the conclusion that the use of inappropriate methods or description of methods without sufficient detail leads to significant biases in the results of CT. Of greatest interest, biases in the distribution of patients between different arms of a CT resulted in an increased effectiveness of treatments of more than 30%.1 These problems led to the formation of the Con solidated Standards of Reporting Trials (CONSORT) group to improve the quality of publications on CT. The CONSORT group made a series of recommendations in 19962 and 20013,4 that were quickly supported by numerous biomedical journals and editorial groups, including the International Committee of Medical Journal Editors, also known as the Vancouver Group. A 2010 update has been recently published.5 These recommendations are based on the monitoring of a checklist of 25 variables based on the evidence and a flowchart to be used to publish, review, and evaluate a CT. The statement, the checklist, the diagram, and the detailed explanation of each item are available at www.consort-statement.org. Three studies have shown that the quality of the publications of the results of CTs that have adopted the CONSORT recommendations is higher than for those journals that did not follow the recommendations.6-8 Nevertheless, many journals publish CT articles whose quality is far from optimal. During the analysis, readers realize that important information to understand the article fully is missing. As mentioned above, Moher et al9 and Schulz et al10 showed that an inadequate description of studies was associated with significant bias in the evaluation of the effects of interventions, in some cases overestimating treatment effects. Because the drug under evaluation in a CT will eventually be used on human beings, if the results are erroneous (for whatever reason), a less effective or totally ineffective treatment than originally proposed could be introduced in daily clinical practice. Therefore, in this study, we comprehensively review the quality of publications of completed CTs conducted at the Hospital Clínico Universitario de Valencia (HCUV) between 2002 and 2005 to determine which sections of the articles should be improved, and we endeavor to provide some guidelines on how to make these improvements. Methods Study Procedure A literature search was performed using the databases PubMed (www.ncbi.nhl.nih.gov/pubmed) and
Pharmacovigilance
Cochrane (www.cochrane.org). The search was limited to completed CTs conducted at the HCUV between 2002 and 2005. The following keywords were used to refine our search: name of investigator AND clinical trial AND tested drug. The search was limited to January 1, 2002, to December 31, 2008. Evaluation of Publications Resulting From Clinical Trials As mentioned above, the CONSORT statement and its subsequent reviews have had a major impact on the quality of publications resulting from CTs. Based on these standards, the published CONSORT checklist was used to evaluate the publications derived from each CT. The list consists of 46 items derived from the CONSORT standards (Table I). Two authors independently evaluated the articles to assess the reporting quality of each item. Reviewers were blinded to CT authors’ identity. A score of 1 was assigned to each item described correctly in the publication and a 0 otherwise. The results of each investigator were collected separately using Microsoft Excel computer software (Microsoft, Redmond, Washington). Analysis of Data To analyze the level of agreement between the 2 reviewers’ ratings, Cohen kappa (κ) index analysis was performed.11 This κ index is a statistical method to analyze the agreement between 2 observers that classify N objects into C mutually exclusive categories. If the agreement is complete, κ = 1, whereas if there is no agreement, κ ≤ 0. A κ > 0.8 was considered an almost perfect agreement; 0.6 to 0.8, considerable; 0.4 to 0.6, moderate; 0.2 to 0.4, fair; and 0.8) in 21 (45%), significant (κ = 0.6-0.8) in 12 (25%), moderate (κ = 0.4-0.6) in 7 (16%), and weak (κ = 0.2-0.4) in 6 (14%) of the indicators. The overall quality of the CTs assessed by CONSORT criteria varied between 20 and 42 (median = 30) out of a possible score of 46. With regard to the methods section, results varied between 5 and 16 (median = 9) out of a maximum score of 17. Within methods subsections, sample size (min = 0, max = 2 of a maximum possible score of 2; median = 0) and the randomization/ blinding (min = 0, max = 4 of a maximum possible score of 5; median = 1) showed the lowest scores. In the results section, scores varied between 2 and 13 (median = 11) of a maximum possible score of 14. In this case, the scores of 2 subsections (basal data and results) showed relatively high scores (median scores of 8 and 3, respectively, of maximum possible scores of 11 and 3, respectively). Sections for the title, abstract, introduction, and discussion showed
In this study, we comprehensively searched the PubMed and Cochrane databases to identify and evaluate the quality of the publications resulting from the CTs conducted at the HCUV between 2002 and 2005. Only 40 articles (of 127 completed CTs) were published between January 1, 2002, and December 31, 2008, which means that only 31% of completed CTs were published. Our results are in agreement with similar studies performed in Spain. In fact, in a study carried out at the Hospital Clinic Barcelona analyzing 123 CTs, Pich et al12 reported that only 27% of started CTs were presented at scientific meetings, and only 31% of closed CTs were published or in press in peer-reviewed journals. In this regard, the CREC should play an important role by ensuring that the results of clinical trials are published. Several reasons have been reported for not publishing a CT. In a review article, von Elm et al13 concluded that the main reason was lack of time, but some researchers also responded that negative results were the major cause for nonpublication. The selective publication of “statistically significant” results is well documented, and it has been observed that a study was more likely to be published if the results were positive.14,15 The high percentage of CT whose promoter is the pharmaceutical industry (100%) should be a point of concern. The large amount of resources required to perform a CT on a large number of individuals has led to the situation that very few CTs are promoted by medical doctors themselves or by the administration. This results in databases that are in the hands of pharmaceutical
1110 • J Clin Pharmacol 2012;52:1106-1114
EVALUATING THE QUALITY OF PUBLICATIONS USING CONSORT GUIDELINES Table III Report Quality Scores for Each Section According to Consolidated Standards of Reporting Trials (CONSORT) Statement Section
Title Abstract Introduction Methods Protocol Sample size Randomization/blinding Statistics Results Basal data Results Discussion CONSORT
Maximum Possible
Maximum Obtained
1 6 3 17 8 2 5 2 14 11 3 5 46
1 5 3 16 8 2 4 2 13 10 3 5 42
Median (Q1,Q3)
1 4 3 9 6 0 1 2 11 8 3 4 30
(0,1) (3,4) (3,3) (8,10) (6,7) (0,0) (1,1) (2,2) (9,12) (8,9) (1,3) (4,5) (29,32)
Minimum Obtained
Mean
SD
0 1 2 5 3 0 0 1 2 0 1 3 20
0.5 3.6 2.8 9.1 6.6 0.1 1.1 2.0 10.2 7.9 2.3 4.3 30.6
0.5 0.8 0.4 1.9 1.2 0.4 0.7 0.2 2.2 1.9 0.9 0.7 3.6
Bold values represent those items that need improvement. Q1, first quartile; Q3, third quartile.
Table IV Weaknesses Observed in the Methods Section of Clinical Trials Published in the Top 4 Journals CONSORT
Missing/Incomplete Information
Sample size
Method used to determine sample size
Randomization
Type of randomization, details of any restriction, implementation
Number of Studies
7 5 3 1 8 5 3 1
Name of the Journal
New England Journal of Medicine Journal of Clinical Oncology Gut Lancet New England Journal of Medicine Journal of Clinical Oncology Gut Lancet
CONSORT, Consolidated Standards of Reporting Trials.
companies and, therefore, scientific dissemination of the results is addressed by the company’s medical department, often without the direct participation of the researchers. This can result in uncontrolled scientific publications and potentially a benefit to the pharmaceutical company.16,17 Several studies have concluded that publicly funded CTs were more likely to be published than privately funded studies.18-20 One possible explanation for this is that the pharmaceutical industry does not tend to disseminate negative results of the CTs that they have funded.21-23 In our study, we did not examine this factor bias because evaluation of unpublished research conducted years ago is a difficult task (interviewing the CT principal investigator, obtaining the necessary information), which is beyond the scope of this study.
Pharmacovigilance
However, it is important to mention that the Spanish legal system through the Royal Decree 223/2004 and Law 29/2006 establishes the requirements for conducting clinical drug trials. In compliance with the legal regulations, the promoter of a CT is required to publish the results (both positive and negative) in scientific journals, with reference to the CREC that approved the study. It further states that the author must mention the proceeds and the funding source.24,25 It has long been known that only a small proportion of CTs are published in scientific journals, which seems to be a common problem because the percentage of publication of CTs in different countries varies between 31% and 67%.26 In our study, the overall quality of the articles according to the CONSORT standards varies
1111
DASÍ ET AL substantially among the publications (scores of 20 and 42 out of a possible maximum of 46), which indicates that some aspects of the articles must be improved. The methods section shows the lowest scores and is therefore of a lower quality providing insufficient and inadequate information. Particularly, the methods used to calculate the sample size and for randomization and blinding are not properly detailed. In fact, although all articles reported on the number of patients, details about the method used to calculate the sample size (power calculation) are described in only 1 article. Reporting of an a priori sample size calculation is an indicator of adequate trial planning and indicates whether a trial ends earlier than planned. Similarly, reporting of blinding was inadequate; most of the articles do not provide detailed information on blinding without indicating who was blinded during the trial. The method of random sequence generation was indicated in just a fifth of the publications. It has been reported than inadequate or unclear reporting of methods of allocation is associated with 30% of larger effect estimates.27 Therefore, there is still considerable room for improvement in the publications of CTs conducted at the HCUV. In particular, the publication of methods described in as much detail as possible and the discussion on the limitations and strengths of the study should be improved. This issue is particularly important because it has been reported that studies with lower methodological quality are most likely to produce “positive results” and are therefore more likely to be published.28 In our study, all the articles reported that the CTs had been approved by a CREC and informed consent had been obtained from the patients. In our opinion (and that of others), the methodological quality is the first ethical requirement of a CT.29 Approval of the CT by a CREC and obtaining the informed consent from patients should be 2 mandatory conditions to carry out a CT and could, therefore, be considered the main components of ethical research in human beings.30 Therefore, both ethical and methodological aspects should be listed in detail in articles, which would help the reader to make a proper assessment of a CT.31 Several studies have concluded that the highest quality CTs give great importance to the ethical aspects of research with humans and, in general, are published in journals of higher impact. This is important because many researchers use the articles published in these journals as examples of how to do good research. Regardless of whether the ethical aspects are specified
1112 • J Clin Pharmacol 2012;52:1106-1114
in the article, they must be fulfilled. Despite this, several studies have reported significant deficiencies in the description of ethical aspects in articles and, in some cases, have been omitted altogether.32,33 A total lack of information on economic studies is observed in our study. This economic information is particularly important in a medical system in which many clinical decisions rely on tools such as the economic evaluation and the intervention outcome. For these reasons, although a section on economy in the CONSORT statement is not explicitly included, we believe that any publication of a CT should include information on the pharmacoeconomics of the procedure. In conclusion, the main reason for carrying out a CT is to test the effect of a drug or a medical procedure and thus improve existing medical practice. Clearly, it contributes most when it is rigorously conducted and the results are published adequately. However, without detailed information on the procedures used in the design, randomization, blinding, or collection and analysis of the data obtained, the validity of a CT is questionable. Despite that, many journals publish CT articles of an unsuitable quality. The lack of adequate reporting led to the development of the original CONSORT guidelines to improve reporting and transparence of CT. Over the past 15 years, a number of CONSORT recommendations (including updates and extensions) for the publication of CTs have been developed. Several reports have demonstrated that the CONSORT statement is an adequate tool to report CT results more accurately. In a recent study, Hopewell and colleagues34 stated that the quality of CT reporting has improved since the publication of the CONSORT guidelines in 2001. In view of these results, one wonders why, if using the CONSORT standards improves the quality of CT reports, many researchers do not use them when publishing their CT results. Several reasons have been proposed, although they all fall within the realm of speculation. Some authors have postulated that the publication guidelines are too prescriptive, limit the creativity of research, and may lack authorization of the self-selected author groups who write them. Lack of awareness of the CONSORT recommendations has also been reported.35 In this sense, journal editors and referees should play an essential role and advocate using the CONSORT recommendations, requiring the authors to follow the guidelines as a condition of publication, which would enhance the quality of the published data and allow reliable assessment of published CTs.36
EVALUATING THE QUALITY OF PUBLICATIONS USING CONSORT GUIDELINES Study Limitations Because of the rigor with which this systematic review was performed, we have full confidence in the information provided. However, the study has several limitations that we would like to point out. Our study examines only the quality of the articles of the CT and never the quality of the CT itself. Several authors have evaluated the methodology of the CT and have concluded that a poor publication does not necessarily mean a bad CT.37 Furthermore, our study focuses on articles published in journals that are indexed within the PubMed and Cochrane databases; if articles were published in other journals that are not indexed, they were not evaluated. In particular, conference papers are difficult to locate as they appear in conference proceedings, which are not usually found in the above-mentioned databases. We are aware, therefore, that a percentage of CTs may have been published in this kind of media and have not been included in this study. However, as previously reported, just half of the studies presented as conference papers are published as original research papers.38 The authors thank Ms Marilyn Noyes for carefully reviewing the manuscript and for the excellent administrative assistance. Financial disclosure: None of the authors has a conflict of interest. Francisco Dasí was funded by the Generalitat Valenciana Grant Number GE-011/09. Esteban Morcillo was funded by the Generalitat Valenciana Grant Number Prometeo/2008/045. Juan R. Viña was funded by the Generalitat Valenciana Grant Number Prometeo/2010/075.
References 1. Moher D, Altman DG, Schultz KF, Elbourne DR. Opportunities and challenges for improving the quality of reporting clinical research: CONSORT and beyond. CMAJ. 2004;171:349-350. 2. Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting randomized controlled trials: the CONSORT statement. JAMA. 1996;276:637-639. 3. Moher D, Schulz KF, Altman DG, for the CONSORT Group. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001;357:1191-1194. 4. Altman DG, Schulz KF, Moher D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001;134:663-694. 5. Schulz KF, Altman DG, Moher D, Fergusson D. CONSORT 2010 changes and testing blindness in RCTs. Lancet. 2010;375:1144-1146. 6. Moher D, Jones A, Lepage L. Use of CONSORT statement and quality of reports of randomised trials: a comparative beforeand-after evaluation. The CONSORT Group. JAMA. 2001;285: 1992-1995. 7. Egger M, Jüni P, Bartlett C. Value of flow diagrams in reports of randomized controlled trials. The CONSORT Group. JAMA. 2001; 285:1996-1999.
Pharmacovigilance
8. Devereaux PJ, Manns BJ, Ghali WA, Quan H, Guyatt GH. The reporting of methodological factors in randomized controlled trials and the association with a journal policy to promote adherence to the Consolidated Standards of Reporting Trials (CONSORT) checklist. Control Clin Trials. 2002;23:380-388. 9. Moher D, Pham B, Jones A, Cook DJ, et al. Does the quality of reports of randomized trials affect estimates of intervention efficacy reported in meta-analysis? Lancet. 1998;352:609-613. 10. Schulz KF, Chlamers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273:408-412. 11. Landis JR, Koch GG. An application of hierarchical kappatype statistics in the assessment of majority agreement among multiple observers. Biometrics. 1977;33:363-374. 12. Pich J, Carné X, Arnaiz JA, Gómez B, Trilla A, Rodés J. Role of research ethics committee in follow-up and publications of results. Lancet. 2003;361:1015-1016. 13. von Elm E, Costanza MC, Walder B, Tramer MR. More insight into the fate of biomedical meeting abstracts: a systematic review. BMC Med Res Methodol. 2003;3:12. 14. Ioannidis JP. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA. 1998;279:281-286. 15. Johnson RT, Dickersin K. Publication bias against negative results from clinical trials: three of the seven deadly sins. Nat Clin Pract Neurol. 2007;11:590-591. 16. Lexchin J, Bero LA, Djulvegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003;326:1-10. 17. Mannhemer E, Anderson D. Survey of public information about ongoing clinical trials funded by industry: evaluation of completeness and accessibility. BMJ. 2002;325:528-531. 18. Dickersin K, Min Y, Meinert CL. Factors influencing publication of research results. JAMA. 1992;267:374-378. 19. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet. 1991;337:867-872. 20. Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997;315:640-645. 21. Strengthening the credibility of clinical research [editorial]. Lancet. 2010;375:1225. 22. Tanne JH. GSK is accused of trying to suppress editorial on rosiglitazone. BMJ. 2010;19:340:c2654. doi:10.1136/bmj.c6254. 23. Cohen D. Rosiglitazone: what went wrong? BMJ. 2010; 6:341: c48448. doi:10.1136/bmj.c4848. 24. Real Decree 223/2004, 6th February for regulating clinical drug trials. Boletín Oficial del Estado nº. February 7, 2004;33:5429-5443. 25. Law 29/2006, 26th July of guarantees and rational use of drugs and health products. Boletín Oficial del Estado nº. July 27, 2006;178:28122-28165. 26. Dwan K, Altman DG, Arnaiz JA, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008;3(8):e3081. 27. Chan AW, Altman DG. Epidemiology and reporting or randomised trials published in PubMed journals. Lancet. 2005; 365:1159-1162. 28. Juni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ. 2001;323:42-46.
1113
DASÍ ET AL 29. Jadad AR, Rennie D. The randomized controlled trial gets a middle-aged checkup. JAMA. 1988;279:319-320. 30. Altman DG. Better reporting of randomised controlled trials: the CONSORT statement. BMJ. 1996;313:570-571. 31. Ruiz-Canela M, Aguinaga I, Guillen F, Martínez-González MA. Publicación de ensayos clínicos: método y ética. Medicina Clínica (Barcelona). 1998;110:238. 32. Ruiz-Canela M, Martínez-González MA, Gómez-Gracia E, Fernández-Crehuet J. Informed consent and approval by institutional review boards in published clinical trials [erratum in N Eng J Med. 1999;341:460]. N Eng J Med. 1999;340:1114-1115. 33. Merz JF, Leonard DG, Miller ER. IRB review and consent in human tissue research. Science. 1999;283:1647-1648. 34. Hopewell S, Dutton S, Yu LM, Chan A, Altman DG. The quality of reports of randomised trials in 2000 and 2006: comparative
study of articles indexed in PubMed. BMJ. 2010;340:c723. doi:10.1136/bmj.c723. 35. Antes G. The new CONSORT statement. BMJ. 2010;340:c1432. doi:10.1136/bmj.c1432. 36. Hopewell S, Altman DG, Moher D, Schulz KF. Endorsement of the CONSORT Statement by high impact factor medical journals: a survey of journal editors and journal ‘Instructions to Authors’. Trials. 2008;9:20. 37. Soares HP, Daniels S, Kumar A, et al. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ. 2004;328:22-24. 38. Scherer RW, Langenberg P, von Elm E. Full publication of results initially presented in abstracts. Cochrane Database Syst Rev. 2007;(2):MR000005.
For reprints and permission queries, please visit SAGE's Web site at http://www.sagepub.com/journalsPermissions.nav.
1114 • J Clin Pharmacol 2012;52:1106-1114
