The EU-PSI Project. Objective: .... Statistical analyses were performed in SPSS (SPSS, Inc., Chi- cago). ... Of 200 randomly selected Psi-Tri trial entries, we ex-.
Article
Safety Reporting in Randomized Trials of Mental Health Interventions Panagiotis N. Papanikolaou, M.D. Rachel Churchill, M.Sc. Kristian Wahlbeck, M.D., Ph.D. John P.A. Ioannidis, M.D. The EU-PSI Project
Objective: The authors aimed to evaluate the adequacy of the reporting of safety information in publications of randomized trials of mental-health-related interventions. Method: The authors randomly selected 200 entries from the PsiTri registry of mental-health-related controlled trials. This yielded 142 randomized trials that were analyzed for adequacy and relative emphasis of their content on safety issues. They examined drug trials as well as trials of other types of interventions. Results: Across the 142 eligible trials, 103 involved drugs. Twenty-five of the 142 trials had at least 100 randomly chosen subjects and at least 50 subjects in a study arm. Among drug trials, only 21.4% had adequate reporting of clinical adverse events, and only 16.5% had adequate reporting of laboratory-determined toxicity,
while 32.0% reported both the numbers and the reasons for withdrawals due to toxicity in each arm. On average, drug trials devoted 1/10 of a page in their results sections to safety, and 58.3% devoted more space to the names and affiliations of authors than to safety. None of the trials of nondrug interventions had adequate or even partially adequate reporting of either clinical adverse events or laboratory-determined toxicity. In multivariate modeling, long-term trials and trials conducted in the United States devoted even less space to safety, while schizophrenia trials devoted more space to safety than did trials in other areas. Conclusions: Safety reporting is largely neglected across trials of mental-healthrelated interventions, thus hindering the assessment of risk-benefit ratios for rational decision making in mental health care. (Am J Psychiatry 2004; 161:1692–1697)
D
etermining adverse events and their frequency for various pharmacological and other interventions is often a challenge (1–4). Randomized trials offer a prime opportunity for collecting not only efficacy data but also useful information on adverse events (1–3, 5, 6). However, empirical evaluations have shown that the reporting of safety information, including withdrawals due to toxicity, clinical adverse events, and laboratory-determined toxicity, is often neglected in randomized controlled trials of therapeutic and preventive interventions (7–11). These deficiencies have serious consequences, limiting the ability to understand the risk-benefit ratios of these interventions (12), even when they prove to be effective. Previous work on safety reporting has addressed trials in various medical fields (7–11). There is evidence that some deficiencies are more prominent in trials from some medical areas than in others (7). Nevertheless, to our knowledge, none of the previous evaluations has specifically targeted trials of interventions in the area of mental health. In this work, we have evaluated safety reporting in a large random sample of randomized controlled trials on various mental-healthrelated interventions.
1692
http://ajp.psychiatryonline.org
Method Trial Database We used a random sample of trials derived from the PsiTri registry. PsiTri is a comprehensive registry of controlled trials across the field of mental health (13). It has been developed as a concerted action funded by the European Commission Quality of Life Programme, and it combines the specialist registries of five mental-health-related collaborative review groups of the Cochrane Collaboration that are working on depression, anxiety, and neurosis; developmental, psychosocial, and learning problems; dementia and cognitive improvement; schizophrenia; and abuse of drugs and alcohol. The PsiTri registry is based on trials rather than references; i.e., references of different reports that pertain to the same controlled trial are grouped under the same trial entry. In December 2002, when sampling was performed, the registry contained 16,504 controlled trials. The registry is evolving, with the continuous addition of more trials, but it is estimated that over 90% of trials in the field are already included. We randomly selected 200 controlled trial entries from the registry. We stratified the random selection by collaborative review group (corresponding to the respective mental health diagnoses) to ensure that the proportion of diseases in the selected sample is representative of the overall registry. We then excluded trials that were controlled but not randomized and trials without peer-reviewed journal publications because it is impossible to access the coverage of safety information simply from abstracts. We included all trials, regardless of sample size. Whenever there were several peer-reviewed publicaAm J Psychiatry 161:9, September 2004
PAPANIKOLAOU, CHURCHILL, WAHLBECK, ET AL. tions on the same trial, we decided which was the main trial publication (the one presenting the most complete version of data on the primary outcomes of the trial) and examined whether the other reports offered any additional information on safety parameters. Both the main report and any additional published information were considered in the appraisal of the adequacy of safety reporting. Whenever a single report presented two or more trials with distinct, independent randomizations, we considered each trial separately.
Parameters of Safety Reporting We used both qualitative and quantitative components of adverse event reporting that have been previously validated (7, 8). Qualitative components include 1) whether the number of withdrawals and discontinuations of study treatment due to toxicity is reported, 2) whether the number is given for each specific type of side effect leading to withdrawal, and 3) whether the severity of the described clinical adverse events and abnormalities of laboratory tests (laboratory-determined toxicity) is adequately defined, only partially defined, or inadequately defined. Adequate definition requires either detailed description of the severity or reference to a known scale of toxicity severity (typically with grades defined as 1=mild, 2=moderate, 3=severe, 4=life-threatening), with at least separate reporting of severe or higher-grade events; at least two adverse events (clinical or laboratory) have to be defined in this way, with numbers or rates given per study arm. Partial definition means that the reported numbers or rates mixed grade 2 with higher toxicity counts or the number of severe or higher grades of toxicity cases are separately specified only for one of many reported adverse events and laboratory abnormalities. Inadequate definition includes protocols lumping numbers of various types of adverse events; those lumping numbers for all grades of toxicity without separating any grades for any specific adverse events; those providing only generic statements (e.g., “the medication was not well tolerated”); and those not reporting anything regarding harm. Quantitative measures assess the relative emphasis given to safety in the results of published trials (7). These measures are 1) the extent of space (absolute numbers of printed pages and proportion) devoted to safety in the results sections; 2) pages devoted to safety compared with pages devoted to the names and affiliations of authors, participants, and contributors in the same trial report; and 3) the use of tables and/or figures containing safety information. The space for each section was measured with a 0.05-page resolution (7). Data were extracted by one investigator (P.N.P.) and discussed with a second investigator (J.P.A.I.) whenever any information was deemed unclear. For the assessment of the adequacy of reporting of clinical adverse effects and laboratory-determined toxicity, the kappa coefficients between independent data extractors have been previously estimated to be 0.72 and 0.85, respectively (8). In a sample of 60 trials, there were no instances in which one extractor considered the reporting adequate and the other inadequate. Moreover, we observed no important discrepancies in data extraction of quantitative parameters.
Other Trial Parameters Trials were categorized first according to whether drugs were involved in the randomized comparisons or not. Nondrug interventions (e.g., psychological, behavioral, and social) may need different approaches to safety. We also separated trials that had at least 100 randomized subjects and over 50 subjects in a study arm. This group has been specifically targeted in previous evaluations of safety reporting (7, 8). Furthermore, for each eligible trial, we noted whether the trial showed significant differences (p
