Correspondence
IC is one of the editors of a 1989 book about systematic reviews, the former editor of the Oxford Database of Perinatal Trials, and has written and spoken about the scientific, ethical, and economic importance of systematic assessment of existing primary research before undertaking additional primary research. PG is the author of many systematic reviews and a textbook about how to do systematic reviews.
*Iain Chalmers, Paul Glasziou
[email protected] James Lind Initiative, Oxford OX2 7LG, UK (IC); and Centre for Research in Evidence Based Practice, Bond University, Robina, QLD, Australia (PG) 1 2
Roberts I, Ker K. How systematic reviews cause research waste. Lancet 2015; 386: 1536. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408–12.
www.thelancet.com Vol 387 January 9, 2016
3
4
5
6
Savović J, Jones HE, Altman DG, et al. Influence of reported study design characteristics on intervention effect estimates from randomized controlled trials. Ann Intern Med 2012; 157: 429–38. Chalmers I. Role of systematic reviews in detecting plagiarism: case of Asim Kurjak. BMJ 2006; 333: 594–96. Ker K, Edwards P, Perel P, Shakur H, Roberts I. Effect of tranexamic acid on surgical bleeding: systematic review and cumulative metaanalysis. BMJ 2012; 344: e3054. Glasziou P, Chalmers I. How systematic reviews can reduce waste in research. Oct 29, 2015. http://blogs.bmj.com/bmj/2015/10/29/howsystematic-reviews-can-reduce-waste-inresearch/ (accessed Dec 4, 2015).
Whether or not all available trials should be included in a metaanalysis has been debated for many years.1 Ian Roberts and Katharine Ker2 argue that systematic reviews increase waste in research by promoting underpowered trials.We disagree and argue that a systematic review is not wasteful if it includes small studies; underpowered trials might be acceptable if investigators use methodological rigor to eliminate bias, thoroughly report results to avoid misinterpretation, and always publish results to avoid publication bias.3 We acknowledge that poorly prepared systematic reviews (eg, reviews that include small trials of poor internal validity) do not minimise waste, but might add to research waste. However, an inadequately prepared randomised trial does not exclude the need for randomised clinical trials; it only highlights the need for clinical trials that are better prepared. The Evidence-Based Research Network,4 which was initiated in 2014 and was inspired by numerous publications,5–7 promotes the notion that to reduce waste in research, new studies should not be done without prior systematic review of existing evidence. Unfortunately, evidence shows that researchers often fail to use earlier research when preparing new research, thereby increasing waste.5–7 Dechartres and colleagues 1 showed that estimation of treatment outcomes in meta-analyses differs depending on the inclusion strategy used.
Experienced methodologists agree that funnel plots, stratified analyses, and sequential analyses of heterogeneityadjusted trial, provide clear insight into the trustworthiness of the different stages of meta-analysis.8 We therefore suggest a new definition of a systematic review, which addresses the concerns of Roberts and Ker about waste: a systematic review is a structured and preplanned synthesis of original studies that consists of predefined research questions, inclusion criteria, search methods, selection procedures, quality assessment, data extraction, and data analysis. No original research study should be deliberately excluded without explanation, and the results from each study should justify the conclusion. Thus, we propose a slightly changed— and definitely more complex— conclusion to the letter by Roberts and Ker: preparation of a systematic review without prior systematic review of existing meta-research is likely to be wasteful, and any meta-analyses that ignore variability in findings without underlining the need for systematic sensitivity analyses are also likely to be wasteful.
Monalyn Gracia/Corbis
have on occasion exposed previously unsuspected research misconduct.4 Important contributions to methodological research have been made through systematic reviews done by Roberts, Ker, and colleagues,5 so it surprises us that they now seem to ignore the fact that their research was based on systematic reviews that included studies of varying methodological quality and size.1 We disagree with their suggested solution for specific clinical questions— to outlaw small trials done in single centres and instead only fund and approve large-scale studies. This recommendation addresses only one of at least four ways in which systematic reviews can inform decisions about whether and which additional primary research should be done.6 Not only is their suggestion scientifically flawed, it is also unrealistic. Funders and regulators cannot be expected to support and endorse large studies without some reassurance from the results of smaller existing studies that the substantial investment needed is justified. We hope that, instead of promoting their one-size-fits-all recommendation, Roberts and Ker will continue to contribute to the efforts needed to improve the quality of primary research and systematic reviews, and improve understanding of how best to use limited resources for research.
HL and RC are members of the Evidence-Based Research Network (EBRNetwork) steering group, and HL is the chairman of the EBRNetwork. RC is involved in health-care initiatives and research associated with Cochrane, Outcome Measures in Rheumatology, International Dermatology Outcome Measurers, Danish Council for the Use of Expensive Hospital Medicines, and the Grading of Recommendations Assessment, Development and Evaluation working group. CJ declares no competing interests.
*Hans Lund, Carsten Juhl, Robin Christensen
[email protected] Department of Sports Science and Clinical Biomechanics, Research Unit for Musculoskeletal Function and Physiotherapy, University of Southern Denmark, 5230 Odense, Denmark (HL, CJ); Centre for Evidence-Based Practice, Bergen University College, Bergen, Norway (HL); Department of Rehabilitation, Herlev and Gentofte Hospital, Copenhagen, Denmark (CJ); and Musculoskeletal Statistics Unit, The Parker Institute, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark (RC) 1
Dechartres A, Altman DG, Trinquart L, Boutron I, Ravaud P. Association between analytic strategy and estimates of treatment outcomes in meta-analyses. JAMA 2014; 312: 623–30.
123
Correspondence
2
3
4
5
6
7
8
See Online for appendix
Roberts I, Ker K. How systematic reviews cause research waste. Lancet 2015; 386: 1536. Schulz KF, Grimes DA. Sample size calculations in randomised trials: mandatory and mystical. Lancet 2005; 365: 1348–53. Chalmers I, Nylenna M. A new network to promote evidence-based research. Lancet 2014; 384: 1903–04. Clarke M, Hopewell S, Chalmers I. Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting. Lancet 2010; 376: 20–21. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet 2009; 374: 86–89. Robinson KA, Goodman SN. A systematic examination of the citation of prior research in reports of randomized, controlled trials. Ann Intern Med 2011; 154: 50–55. Nüesch E, Jüni P. Commentary: Which meta-analyses are conclusive? Int J Epidemiol 2009; 38: 298–303.
In their letter, 1 Ian Roberts and Katharine Ker made three assumptions: that small clinical trials are all of poor quality; that all large, multicentre, well funded trials are of exemplary quality; and that meta-analysis cannot ameliorate the low power of small trials. None of these assumptions are documented. Although many clinical trials do have methodological flaws, many do not, and this issue is addressed in systematic reviews with sensitivity analysis. Certainly, the aims of meta-analysis are to achieve increased statistical power and better quality assessment and to obtain a summary statistic. An excellent example of the not infrequent shortcomings of large, well funded trials is the breast cancer screening study 2 published in The Lancet, in which 75% of randomised trials were excluded from meta-analysis because of considerable quality issues. I declare no competing interests.
Richard L Nelson
[email protected] School of Public Health, University of Illinois at Chicago, Chicago, IL 60612, USA 1 2
124
Roberts I, Ker K. How systematic reviews cause research waste. Lancet 2015; 386: 1536. Gøtzsche PC, Olsen O. Is screening for breast cancer with mammography justifiable? Lancet 2000; 355: 129–34.
Oseltamivir for influenza Previously1 we called on Joanna Dobson and colleagues2 to make public the protocol for their meta-analysis of randomised controlled trials of oseltamivir. This document could easily be uploaded to a repository such as Dryad, and linked to their research paper. We are alarmed that they neither did this nor explained their reasons for not doing so.3 We are taking this opportunity to reiterate our request. For their work to be taken seriously, there is an imperative to furnish the scientific community with a protocol that pre-dates their analysis. Numerous decisions, from selection of dose for examination, to prespecified and exploratory analysis conducted, definitions of outcomes examined, and statistical methods chosen, cannot be understood and critically assessed without access to the protocol. The Lancet REWARD initiative underscores how important such documents are to sound science.4,5 Furthermore, the Roche-funded Multi-party Group for Advice on Science (MUGAS), which funded Dobson and colleagues’ analysis, held a one day meeting in June 2013 “to develop a statistical analysis plan on Oseltamivir data, both from randomized controlled trials and from observational trials.” The MUGAS press release stated: “This plan will serve as a basis for the post-meeting in depth statistical analysis, eventually to be carried out by MUGAS and MUGAS invited partners”. We call on the authors to release all of these pre-trial documents, and we call on The Lancet to join us in this demand and consider the ramifications of the documents continuing to remain secret. In addition, the conflict of interest disclosure statements for Dobson and colleagues’ original article2 and letter3 are incomplete and need to be corrected. Dobson and Stuart Pocock
declare “no competing interests”, but Dobson’s salary was supported by Roche during the analysis and Pocock told a journalist that he received research funds from Gilead and Genentech.6 Also, Richard Whitley does not inform readers that Gilead Sciences is the patent holder for oseltamivir so readers can evaluate the implications. According to ProPublica and OpenPaymentsData.CMS.gov, Arnold Monto and Whitley also both received money in recent years from companies not listed in The Lancet articles. Finally, despite the fact that all authors are financially linked to the manufacturer or patent holder of oseltamivir, the linked editorial by Heath Kelly and Benjamin Cowling7 inaccurately describes Dobson and colleagues as an “independent research group”. The above inaccuracies all need to be addressed, and we call on The Lancet to help ensure that the record is corrected. See appendix for declaration of interests
*Peter Doshi, Carl Heneghan, Tom Jefferson
[email protected] Department of Pharmaceutical Health Services Research, University of Maryland School of Pharmacy, MD, USA (PD); Centre for Evidence-Based Medicine, University of Oxford, UK (CH); and Cochrane Acute Respiratory Infections Group, 00187, Roma, Italy (TJ) 1 2
3
4
5
6
7
Doshi P, Heneghan C, Jefferson T. Oseltamivir for influenza. Lancet 2015; 386: 1134–35. Dobson J, Whitley RJ, Pocock S, Monto AS. Oseltamivir treatment for influenza in adults: a meta-analysis of randomised controlled trials. Lancet 2015; 385: 1729–37. Monto AS, Dobson J, Pocock S, Whitley RJ. Oseltamivir for influenza—Authors’ reply. Lancet 2015; 386: 1135–36. Chan A-W, Song F, Vickers A, et al. Increasing value and reducing waste: addressing inaccessible research. Lancet 2014; 383: 257–66. Ioannidis JPA, Greenland S, Hlatky MA, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 2014; 383: 166–75. Lenzer J. Why aren’t the US Centers for Disease Control and Food and Drug Administration speaking with one voice on flu? BMJ 2015; 350: h658. Kelly H, Cowling BJ. Influenza: the rational use of oseltamivir. Lancet 2015; 385: 1700–02.
www.thelancet.com Vol 387 January 9, 2016
