pharmacoepidemiology and drug safety 2014; 23: 849–858 Published online 19 June 2014 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/pds.3669
ORIGINAL REPORT
Signal detection based on time-to-onset: extending a new method from spontaneous reports to observational studies Lionel Van Holle*,†, Fernanda Tavares Da Silva† and Vincent Bauchau† Vaccines Clinical Safety and Pharmacovigilance, GlaxoSmithKline Vaccines, Wavre, Belgium
ABSTRACT Purpose A proof-of-concept study has previously highlighted the added value of a method using time-to-onset (TTO) for quantitative and non-parametric signal detection on spontaneous report data. The aim of this study was to assess the added value of this new TTO signal detection method adapted to observational studies. Methods For each adverse event collected during the conduct of an observational study of H1N1 pandemic influenza vaccine, the TTO distribution was tested against the ‘follow-up distribution’ from vaccination to ‘lost to follow-up’ by a Kolmogorov–Smirnov test. Events rejecting the null hypothesis of similar distribution were flagged as signals, and a safety physician evaluated their relevance for further medical assessment. We simulated ongoing surveillance by performing retrospective weekly signal detection based on TTO. Results The TTO method detected 21, 15 and 4 signals within a 30-day period post-dose 1 with confidence levels set at 90%, 95% and 99%, respectively. Of these signals, 14 (67%), 10 (67%) and 2 (50%) were considered as relevant. Among the 14, six had not been identified by previous signal detection activities. When performed weekly, the Kolmogorov–Smirnov test detected 26 events as signals (alpha = 0.05). Three weeks after first participant first dose, one of the six new signals could theoretically have been detected. Conclusions This study provided evidence that the Kolmogorov–Smirnov method can screen all TTO distributions and objectively flag the unexpected, leading to earlier detection of signals, and thus potential safety issues. © 2014 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons, Ltd. key words—vaccine safety; signal detection; pharmacovigilance; observational study; time-to-onset; Kolmogorov–Smirnov; pharmacoepidemiology Received 20 November 2013; Revised 14 March 2014; Accepted 2 June 2014
INTRODUCTION Real-time safety surveillance during the conduct of observational (or clinical) studies is critical for early detection of unexpected safety problems. Indeed, Directive 2001/20/EC of the European Parliament and of the Council1 states ‘it is necessary to make provision for the monitoring of adverse reactions occurring in clinical trials using community surveillance (pharmacovigilance) procedures in order to ensure the immediate cessation of any clinical trial in which there is an unacceptable level of risk’. For large-sampled studies as in Post Authorization Safety Studies (PASS) for vaccines, an extensive
*Correspondence to: L. Van Holle, GlaxoSmithKline Vaccines, Parc la Noire Epine, Avenue Fleming, 20, 1300 Wavre, Belgium. Email:
[email protected] † All authors state that the manuscript has not been published elsewhere.
medical review of all adverse events (AEs) for assessing the potential causality of the product(s) under study may be challenging. Quantitative signal detection methods are sometimes used for prioritising medical review to the AEs presenting the highest probability of being causally associated with the product under study (e.g. relative risk calculation for all AEs, comparison of their survival distributions2) or for monitoring some AEs of special interest (e.g. with the Maximised Sequential Probability Ratio Test—MaxSPRT3). In case of single-arm study or when treatment groups are blinded, there is currently no real-time quantitative method available in the literature for routinely assessing the causality of the product under study for every observed AE. Indeed, no formal comparison between groups is possible, at least not before unblinding (for blinded studies). In addition, for single-arm studies, a method based on the MaxSPRT could not monitor every AE as it is not realistic to obtain
© 2014 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons, Ltd. This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.
850
l. van holle et al.
reliable estimated background incidence rates for every observed AE and for a population comparable with the study population, which depends on specific inclusion and exclusion criteria. We recently demonstrated the feasibility of signal detection based on the time-to-onset (TTO: the time between vaccine administration and onset of symptoms of an AE) in the context of post-marketing surveillance based on spontaneous reporting.4–6 Here, we have adapted this TTO signal detection method for use in ongoing observational studies, with the aim of meeting the challenge of performing real-time signal detection for every observed AE. A systematic review of the methods for causality assessment of adverse drug reactions7 highlighted that the TTO was the most frequent causality criterion used across the different methods. An empirical cumulative distribution of TTO was already recommended as a visual inspection tool in cohort study design in Germano Ferreira’s thesis.8 It highlights the pertinence of developing signal detection methods based on the TTO in the context of large-sample observational (or clinical) studies. METHODS Null hypothesis Under the null hypothesis of no association between vaccination and the event-of-interest, the number of participants presenting with a first event at time t after vaccination follows a Poisson process. The Poisson process {N(t), t ≥ 0} is a counting process with the following properties:
then the event times T1,…,Tn, are distributed as the order statistics of n independent random variables that are uniformly distributed on [0,t]. The additional assumption underlying this uniform distribution of time T1,…,Tn from vaccination to onset of event is that no individual is lost to follow-up within the interval [0,t]. When it happens, the time distribution from vaccination to the first event should be similar to the follow-up distribution from vaccination to lost to follow-up. This ‘follow-up distribution’ can be seen as the sum of uniform distributions in the [0,t] interval but censored at the individual lost to follow-up times. One-sample Kolmogorov–Smirnov test For each event, the time distribution from vaccination to event was tested against the ‘follow-up distribution’ from vaccination to lost to follow-up within the period [0,t], with a one-sample Kolmogorov–Smirnov (KS) test.10 The KS test is a non-parametric test sensitive to any differences between distributions, such as differences in location, dispersion or skewness. Let Sm(X) be the empirical distribution function for one sample (of size m), that is, Sm(X) = K/m, where K is the number of data equal to or less than X, and let S(X) be the ‘follow-up distribution’ function. The one-sample KS test statistic is D ¼ maxjSm ðX Þ–SðX Þj Graphically, D is the greatest vertical distance between the observed TTO and follow-up distributions (Figure 1).
(1) N(0) = 0. n (2) P(N(t) = n) = eλt ðλtn!Þ , n = 0, 1, 2,…. The Poisson process has the desirable properties of independent and stationary increments;9 thus, the numbers of events in disjoint intervals are independent, and the number of events in any given interval depends only on the length of that interval and not on its position in time. One hidden and reasonable assumption underlying these properties is that the natural incidence of an event is assumed to be constant over the age intervals [Ai, Ai + t] and the periods [Ci, Ci + t], where Ai is the age of participant i and Ci is the calendar time at the time of vaccination of participant i. The lower t is, the more reasonable the assumption is. If one event of a Poisson process has occurred in [0,t], then the time of that occurrence is uniformly distributed over [0, t] (each subinterval of equal length having the same probability of containing that event). If N(t) = n,
D
Figure 1. Illustration of the test statistic D of the Kolmogorov–Smirnov test (the follow-up distribution illustrated here follows a uniform distribution meaning that no study participant was lost to follow-up within 30 days post vaccination)
© 2014 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons, Ltd.
Pharmacoepidemiology and Drug Safety, 2014; 23: 849–858 DOI: 10.1002/pds
signal detection based on tto in observational studies
The null distribution of this statistic is calculated under the null hypothesis that the samples are drawn from the follow-up distribution. As it is not a standard distribution, a Monte-Carlo simulation based on the follow-up distribution was performed to obtain the 99.5, 97.5 and 95 percentiles needed to perform the one-sample KS at confidence levels (CLs) of 99%, 95% and 90%, respectively. Observational study: the UK PASS The UK PASS was a prospective, observational, multicentre, single-cohort post-authorisation safety study designed to detect safety signals.11 The study was sponsored by GlaxoSmithKline Biologicals SA as part of the AS03-adjuvanted H1N1 (2009) vaccine Risk Management Plan. The target population consisted of men and women who received at least one dose of the AS03adjuvanted split virion H1N1 (2009) pandemic influenza vaccine (PandemrixTM) during the national pandemic influenza vaccination campaign in the UK. A total of 9215 participants were enrolled according to recommendations from the Committee for Medicinal Products for Human Use (CHMP) of the European Medicines Agency.12 Data were available for analysis for 9143 participants (study cohort) with 682 participants (52.8% female) included in the reactogenicity analysis. The primary objective of the UK PASS was to estimate the incidence of medically attended AEs (MAEs) in all vaccinated participants within 31 days post-vaccination. The secondary objectives were as follows: to assess vaccine reactogenicity within 7 days after vaccination in different age groups (solicited local and general symptoms); to estimate the incidence of serious AEs (SAEs) and AEs of special interest (AESIs), as defined by the CHMP, occurring within 181 days after any dose; and to monitor all pregnancy outcomes in vaccinated pregnant women and women who became pregnant after vaccination. Reports of MAEs, SAEs and AESIs following vaccination were collected for all participants; solicited AEs were assessed in a reactogenicity subset (600 participants). A total of 445 participants received a second dose. Analysing the performance of time-to-onset signal detection For each MAE, SAE and AESI, the empirical TTO distribution was compared with the ‘follow-up
851
distribution’ from vaccination to ‘lost to follow-up’ with a one-sample KS test for the 30-day period post-dose 1, post-dose 2 and post any dose period. Alpha levels were set at 0.01, 0.05 and 0.10. The different percentiles for the KS distance were obtained by simulation under the null hypothesis that the TTO distribution comes from the same distribution as the follow-up one. Events presenting significantly different TTO distributions from the follow-up distribution were flagged as signals. These signals were submitted to a senior physician responsible for medical safety assessment in the UK PASS and who was blinded to the statistical significance of the signals to avoid biases in the assessment. The physician classified the signals as follows:
• • •
Possible causal association with study vaccine (Yes/No/Unknown). Signal previously investigated during the conduct of the study (Yes/No). Relevant for further medical assessment (Yes/No).
Events flagged as ‘Yes’ for any of the aforementioned criteria were considered as gold standard. Accordingly, signals detected by the KS test were classified as truepositive or false-positive signals. The timing of detection is important as early detection of serious safety concerns would allow potential corrective measures in the study conduct. We therefore investigated the timing of the detection by retrospectively simulating a weekly run of the KS test. For simplicity, we present results post-dose 1 only and until the completion of the 30-day period for every UK PASS participant. The best usage of the one-sample KS test was defined on the basis of the number and proportion of true-positive signals among those detected (also known as positive predictive value). RESULTS Post-dose 1 Table 1 shows the signals identified using different CLs for the one-sample KS test of the TTO distribution of every event in a 30-day period post-dose 1. The KS test detected 21, 15 and 4 signals at 90%, 95% and 99% CLs, respectively (Table 2). Among them, 14 (67%), 10 (67%) and 2 (50%) were considered as true positives by the physician. The additional restriction of a minimal number of three cases would have given a lower number of signals but a higher percentage of true positives.
© 2014 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons, Ltd.
Pharmacoepidemiology and Drug Safety, (2014); 23: 849–858 DOI: 10.1002/pds
852
l. van holle et al.
Table 1. Description of the MedDRA PTs flagged as signals by the one-sample Kolmogorov–Smirnov test for the time-to-onset distribution of any event in a 30-day post-vaccination period MedDRA PT
Number of cases
90% CL signal
95% CL signal
99% CL signal
Causally related
Already investigated
Signal relevant
2 2 14 2 2 3 2 2 16 5 2 2 3 7 4 2 103 3 6 3 2
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 0 1 1 0 1 0 0 1 1 1 1 1 1 1 1 1 0 0
0 0 0 0 0 0 0 1 0 0 0 0 0 1 1 1 0 0 0 0 0
No No Unk No No Unk Yes No Yes Unk Unk Yes Yes Yes Yes No Unk Yes Unk No Unk
No No No No No No Yes No Yes Yes No Yes Yes Yes Yes No No Yes No No No
No No Yes No No Yes Yes No Yes Yes Yes Yes Yes Yes Yes No Yes Yes Yes No Yes
Atrial fibrillation Balanitis Chest pain Contusion Dysphagia Epicondylitis Groin pain Haematuria Headache Hypersensitivity Injection site cellulitis Injection site erythema Injection site rash Injection site reaction Injection site swelling Insomnia Lower respiratory tract infection Malaise Pain Postpartum depression Subdural haematoma
PT, preferred term; Unk, unknown; CL, confidence level.
Table 2. Summary of the MedDRA PTs flagged as signals by the onesample Kolmogorov–Smirnov test for the time-to-onset distribution of any event in a 30-day period post vaccination
Events Any Any Any
Time window length (days)
Confidence level (%)
% of TP (total) > 1 case
% of TP (Total) > 2 cases
30 30 30
90 95 99
67 (21) 67 (15) 50 (4)
91 (11) 100 (8) 100 (2)
PT, preferred term; TP, true positives.
Six MedDRA1 preferred terms (PTs) that had not been detected by previous standard signal detection were flagged as relevant: Chest pain, Epicondylitis, Injection site cellulitis, Lower respiratory tract infection, Pain and Subdural haematoma. Figure 2 shows their TTO distribution in the 30-day post-dose 1 period. Figure 3 shows the follow-up distribution as observed at the completion of the study. It shows a very slight decrease over time as it takes into account the lost to follow-up.
The detection of these six events on a weekly basis post-dose 1 was also estimated (Table 3), with application of the one-sample KS test at 90% and 95% CL using the follow-up distribution as it would have been observed at the weekly time points (Figure 4). Using a 90% CL, 43 events were flagged as signals at least at 1-week time-point. After completion of the 30-day post-dose 1 period by all participants, only 21 of those 43 events were still flagged as signals. Therefore, 22 (51%) of the signals could be considered to be statistical false positives due to multiple testing. Similarly, at 95% CL, 26 events were flagged as signals at least once, whereas only 15 were still signals at the end of the 30-day period post-vaccination by all participants. Eleven (42%) signals could then be considered as false positives. For four of the six signals detected by the TTO signal detection method, but not by previous methods, the timing of their detection was the same, regardless of the CL chosen. For only two events, detection would have occurred 1 week later using a 95% rather than a 90% CL (Table 3). Post-dose 2
1
Medical Dictionary for Regulatory Activities is a clinically validated international medical terminology used by regulatory authorities and the regulated biopharmaceutical industry throughout the entire regulatory process, from pre-marketing to post-marketing activities and for data entry, retrieval, evaluation and presentation.
Only a small percentage of UK PASS participants received a second dose; 1 MedDRA PT (Lower respiratory tract infection) was flagged as a signal, with the KS test at 90% CL for the 30-day post-dose 2 period.
© 2014 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons, Ltd.
Pharmacoepidemiology and Drug Safety, (2014); 23: 849–858 DOI: 10.1002/pds
signal detection based on tto in observational studies
Figure 2.
Time-to-onset distribution of the MedDRA PTs flagged as relevant signals and not detected by standard signal detection methods
Figure 3.
Follow-up distribution over 30 days post dose 1 as observed at the end of the study
© 2014 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons, Ltd.
853
Pharmacoepidemiology and Drug Safety, (2014); 23: 849–858 DOI: 10.1002/pds
854 Table 3.
l. van holle et al. Detection, on a weekly basis, of the six MedDRA PTs found as relevant but not investigated during routine, standard pharmacovigilance activities
MedDRA PT
Chest pain Epicondylitis Injection site cellulitis Lower respiratory tract infection Pain Subdural haematoma Completion of the 30-day period post-dose 1 (%)
Week
CL (%)
90 95 90 95 90 95 90 95 90 95 90 95
1
2
3
4
5
6
7
8
9
10
0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 3
1 1 0 0 0 0 0 0 0 0 0 0 9
1 0 0 0 0 0 0 0 0 0 0 0 21
0 0 0 0 0 0 0 0 1 1 0 0 38
0 0 0 0 1 0 0 0 1 1 0 0 59
1 1 0 0 1 1 1 0 1 1 0 0 77
1 1 0 0 1 1 1 1 1 1 1 1 91
1 1 1 1 1 1 1 1 1 1 1 0 98
1 1 1 1 1 1 1 1 1 1 1 0 100
PT, preferred term; CL, confidence level.
Figure 4. Follow-up distribution over 30 days post dose 1 as it would have been observed at different time points on an ongoing basis
© 2014 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons, Ltd.
Pharmacoepidemiology and Drug Safety, (2014); 23: 849–858 DOI: 10.1002/pds
signal detection based on tto in observational studies
Post-any dose For the cumulative post-dose 1 and post-dose 2 TTO profiles, the number of MedDRA PTs flagged as a signal decreased. ‘Balanitis’, ‘Chest pain’, ‘Headache’, ‘Hypersensitivity’ and ‘Lower respiratory tract infection’ were no longer flagged as signals when the onesample KS test was run at 90% confidence limit for the 30-day period post-any dose. By contrast, ‘Injection site pain’ appeared as a new signal. DISCUSSION The method described here detected a relatively high proportion of events already listed in the reference safety information of the product (events reasonably assumed to be causally associated with the investigational product and part of the known safety profile of the product), including headache, abdominal pain, hypersensitivity, injection site reactions (including erythema, swelling and pain) and also signals that had already been analysed during the study such as respiratory tract infections, pregnancy-related outcomes/ complications and allergic type reactions. Among the six signals that had not been detected during the conduct of the study, two were medically significant events meriting further investigation: injection site cellulitis and subdural haematoma. Furthermore, at the time of the final analysis by standard signal detection methods, the most frequently reported MAEs and SAEs were determined to be associated with ‘infections and infestations’, primarily MedDRA PTs relating to lower and upper respiratory tract infections, which was an anticipated finding in this study population, particularly during the winter season. However, it was very difficult to assess if the number of respiratory tract infections was within the range we could expect. The TTO method appears to be useful in its capability to flag temporal relationships between infections and vaccinations. Quantitative signal detection methods were traditionally focusing on quantifying the strength of association at the population level. The KS signal detection method allows quantification and objective evaluation of the temporal relationship independently from the reviewer perception and from current knowledge in biologic plausibility, which could restrict arbitrarily the list of temporal associations considered as of interest. Therefore, the KS test allows independent quantification of an additional causality criterion at the population level.13 However, the application of any quantitative signal detection applied to a long list of observed events may result
855
in statistical false-positive signals or false-positive signals due to a lack of adjustment making these quantitative signals only potential safety signals. Consequently, the quantitative signals must go through an evaluation of the other causality criteria at both the population and individual levels. The generally established causality criteria at the individual level are as follows: temporal relationship, presence of clinical or laboratory proof, consistency with causality assessment at the population level, likelihood of alternative explanations and prior evidence of a similar mechanism.13 The more concordant the criteria are, the more likely is the presence of an association. Conversely, the more discordant the criteria are, the less likely is the presence of an association. Some tentative for weighting the different causality criteria has been initiated,14,15 but the weights could differ between events and products as different mechanisms could be involved. The use of our method may assist reviewers in prioritising individual case review activities, identifying sets of cases of potential interest and facilitating appropriate prioritisation earlier during the course of the study, rather than at the time of final analysis. Indeed, the most efficient use of the one-sample KS test of the TTO distribution should be considered in two different contexts. The first is at the end of the study, when the post-vaccination period of interest is entirely completed by all study participants. When testing the TTO distribution of any event in a 30-day period post-dose 1 (once completed by all participants), the positive predictive value was always higher than 50%. However, applying a 90% confidence limit with a minimum of two cases seemed to give the optimum balance between having a high number of true-positive signals and a good positive predictive value. The second context to consider is the application of the KS test at periodic time-points. When we performed the TTO KS test on a simulated weekly basis, some false-positive signals were generated by multiple testing. The percentage of false-positive signals seemed to be higher at 90% than at 95% CL. A pragmatic approach would be to use the KS method in both contexts, with periodic application until study completion and a final test at lower CL. In our study, the follow-up distribution at the end of the study could have been approximated by a uniform distribution because of the very low fraction of study participants being lost to follow-up within the 30-day period after dose 1 (Figure 3). However, this approximation can definitely not be made when performing the KS test on real-time data. Indeed, because of continuous enrolment of study participants the follow-up distribution varies greatly over time (Figure 4).
© 2014 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons, Ltd.
Pharmacoepidemiology and Drug Safety, (2014); 23: 849–858 DOI: 10.1002/pds
856
l. van holle et al.
The evaluation of TTO distributions after each dose and then pooled allowed the detection of violations of hidden assumptions. Indeed, the ‘Lower respiratory tract infection’ MedDRA PT was flagged as a signal post-dose 1 and post-dose 2 but was no longer flagged when evaluated for post-any dose. This could be due to seasonal changes in the prevalence of these respiratory infections during the [D1, D1 + 30] and [D2, D2 + 30] intervals, with a lower prevalence during the first half of [D1, D1 + 30] and last half of [D2, D2 + 30]. Indeed, dose 1 was distributed around the beginning of the seasonal peak of these infections, whereas dose 2 was distributed around the end of the peak. We recommend checking the assumptions on which the one-sample KS test is built. For example, we observed that on Sundays, only one participant was vaccinated on average compared with 250 on the other days. Additionally, the starting dates of the events tended to occur less than half as frequently on Sundays than on the other days. This systemic bias in the UK PASS generated a small periodic interference in the time distribution of events that should be observed in theory under the null hypothesis of no association between the vaccination and the event. The theoretical distribution is then in practice also dependent on factors other than the lost to follow-up time. This slightly violates the assumption underlying the Poisson process and its stationary increment property. In theory, the follow-up distribution could be adjusted for that systemic bias before applying the one-sample KS test. However, this exercise was not performed here as we considered it as beyond the scope of this study. This TTO signal detection method based on a onesample KS test is independent of any estimate coming from an external source (the literature) or derived from a control group. It therefore has the following advantages: (1) The KS test can be applied in the absence of control group, as illustrated here. (2) The KS test provides an objective quantification of how unexpected the observed TTO distribution is. (3) The KS test is non-parametric and is sensitive to any difference between the empirical and theoretical distributions, such as differences in location, dispersion or skewness. It provides a major advantage to a parametric approach such as the Weibull shape parameter signal detection tool16 tested on electronic healthcare record data and presenting difficulties detecting variation in hazards when the increased risk appears towards the middle of the observation period.16,17
(4) The KS test can be applied to the full list of events observed in a study without restricting monitoring to a subset of preselected events for which prevalence estimates have been gathered from the literature as for the MaxSPRT method. (5) The KS test can be applied to ongoing studies provided that follow-up in the specified period can be estimated for every participant. (6) The KS test could theoretically be applied to blinded studies: the control group would generate some background noise making the KS test less sensitive to TTO distortions occurring in the product group only. (7) The KS test does not require adjustment by covariates that are time-independent as only the TTO distribution is tested. On the other hand, this TTO signal detection method detects only the AEs presenting an association with the vaccine characterised by a distortion of the TTO distribution within the post-vaccination period under investigation. This method has no power to detect distortions that occur after the period under investigation, or associations that generate an increased risk that remains constant during the period under investigation. The KS test should be considered as a complementary tool, to be used alongside other methods, in routine signal detection during clinical or observational studies. As this method does not require comparative data, we think it could be a promising tool for real-time safety monitoring in blinded or single-arm clinical trials for which the assessment of the strength of association between the vaccine and the events could be difficult as long as treatment groups are blinded and/or there is no comparator. We envision this method to be also applicable to longitudinal electronic health care records. In these observational databases, multiple biases (visit effect, etc.) may impact the distribution of the TTO (even in the absence of any association with the exposure of interest). Therefore, one may not be able to derive an expected (theoretical) distribution of the TTO under the null hypothesis. However, this problem could be overcome by switching from a one-sample KS test to a two-sample KS test. Similar to the strategy taken for spontaneous report data, the empirical distribution observed for other events than the one of interest could be used as comparator under the assumption that there is no association between the vaccine and any of the other events (basically the same assumption that underlies disproportionality analyses). The comparator distribution would capture biases inherent to electronic patient records and the two-sample KS test would
© 2014 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons, Ltd.
Pharmacoepidemiology and Drug Safety, (2014); 23: 849–858 DOI: 10.1002/pds
signal detection based on tto in observational studies
consequently be adjusted for such biases. This adapted method, which does not require other vaccines as comparator, could be an alternative to the chronograph method developed by Norén,18,19 which relies on the presence of a large number of other products for estimating the information component. Indeed, there are far less vaccines than drugs on the market, and it could potentially bias downwards20 the information component if the chronograph should be used for detecting temporal patterns after immunisation in longitudinal electronic records. At this stage, this is only speculative, and to determine respective signal detection performances, an empirical evaluation of both methods for its application to vaccines on longitudinal electronic records should be performed. The Observational Medical Outcomes Partnership could offer a framework for such performance comparison.21 It is anticipated that this method could be applied for detecting temporal associations of an event with any medicinal product as the null hypothesis of no association should also result in the follow-up distribution being used as control for the different TTO distributions tested by the KS test. However, additional research is needed to explore signal detection performance for other types of medicinal products, which, in case of association, are more likely than for vaccines to be subject to complex temporal relationships induced by withdrawal/dechallenge, re-exposure/ rechallenge, dose–response relationship or drug–drug interactions. Furthermore, some diseases such as cancer tend to have chronic evolution and complex triggering mechanisms, and if ever the medicinal product was the trigger, the first sign/symptoms are more likely to be widely distributed over time making it difficult to be detected by a method such as the KS test, which relies solely on the TTO distribution. From purely a statistical perspective, common events with acute onset are more likely to be captured earlier by this method than rare and/or chronic events. This study will allow potential users to be fully aware of the advantages and limitations of this method compared with other methods. CONCLUSION
857
CONFLICT OF INTEREST All authors are employees of GlaxoSmithKline Vaccines and own restricted shares of the company. PandemrixTM is a trademark of the GlaxoSmithKlinegroup of companies.
KEY POINTS
• •
• •
In the absence of control group, there is no standard routine quantitative signal detection method for detecting safety signals in observational studies. Under the null hypothesis of no association between the vaccine and the events, the time distribution from vaccination to event should be similar to the ‘follow-up distribution’ from vaccination to lost to follow-up. The one-sample KS test can be used to test this null hypothesis for every event and can flag events violating this assumption as potential safety signals worth further investigation. This method could add value to other routine safety monitoring methods, leading to additional and/or earlier detection of signals.
ETHICS STATEMENT A poster was presented at the International Conference of Pharmacoepidemiology, Montreal (2013). ACKNOWLEDGEMENTS The authors thank François Haguinet for numerous discussions and support. Andreas Brueckner (Novartis, Basel, Switzerland) reviewed the manuscript. Editing and publication co-ordinating services were provided by Juliette Gray (XPE Pharma & Science, Wavre, Belgium), and Veronique Delpire and Mandy Payne (Words and Science, Brussels, Belgium). GlaxoSmithKline Biologicals SA funded all costs associated with the development and the publishing of the present manuscript. REFERENCES
This study showed that signal detection based on the TTO could add value to other routine safety monitoring methods, leading to additional and/or earlier detection of signals and thus potential safety issues. The KS test is relatively simple to implement, using standard computer software, and can detect timedependent signals. © 2014 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons, Ltd.
1. Directive 2001/20/EC of the European parliament and of the council of 4 April 2001 on the approximation of the laws, regulations and administrative provisions of the member states relating to the implementation of good clinical practice in the conduct of clinical trials on medicinal products for human use. Off J Eur Communities 2001; L121: 34–44. http://eur-lex.europa.eu/LexUriServ/ LexUriServ.do?uri=OJ:L:2001:121:0034:0044:en:PDF [20 November 2013]. 2. Lin X. A new method for the comparison of survival distribution. Pharmaceut Statist 2010; 9: 67–76. 3. Kulldorf M. A maximised sequential probability ratio test for drug and vaccine safety surveillance. Sequential Anal 2011; 30: 58–78.
Pharmacoepidemiology and Drug Safety, (2014); 23: 849–858 DOI: 10.1002/pds
858
l. van holle et al.
4. Van Holle L, Zeinoun Z, Bauchau V, Verstraeten T. Using time-to-onset for detecting safety signals in spontaneous reports of adverse events following immunization: a proof of concept study. Pharmacoepidemiol Drug Saf 2012; 21(6): 603–10. 5. Van Holle L, Bauchau V. Signal detection on spontaneous reports of adverse events following immunisation: a comparison of the performance of a disproportionality-based algorithm and a time-to-onset-based algorithm. Pharmacoepidemiol Drug Saf 2014; 23: 178–185. 6. Karimi G, Star K, Hägg S, Norén GN. Time-to-onset in spontaneous reports: the possibility to detect the unexpected. Pharmacoepidemiol Drug Saf 2013; 22: 556–557. 7. Agbabiaka TB. Methods for causality assessment of adverse drug reactions—a systematic review. Drug Saf 2008; 31(1): 21–37. 8. Ferreira G. Development of methodologies for improvement in signal detection using prescription–event monitoring as a post-marketing surveillance platform [online]. http://ethos.bl.uk/OrderDetails.do?did=1&uin=uk.bl.ethos.494013 [26 November 2012]. 9. Gallager R. 6.262 Discrete Stochastic Processes, Spring 2011. (Massachusetts Institute of Technology: MIT OpenCourseWare) [online]. http://ocw.mit.edu [accessed 27 September 2012]. License: Creative Commons BY-NC-SA. 10. Siegel S, Castellan NJ, Jr. Nonparametric Statistics for the Behavioral Sciences. Statistics Series (2nd edn). McGraw-Hill International Editions: New York, 1988; 51–55. 11. Nazareth I. Safety of AS03-adjuvanted split-virion H1N1 (2009) pandemic influenza vaccine: a prospective cohort study. BMJ Open 2013; 3: e001912. DOI:10.1136/bmjopen-2012-001912. 12. European Medicines Agency. CHMP Recommendations for the Pharmacovigilance Plan as part of the Risk Management Plan to be submitted with the Marketing
13.
14. 15. 16. 17.
18.
19. 20.
21.
© 2014 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons, Ltd.
Authorisation Application for a Pandemic Influenza Vaccine. Revision 1.1. EMEA/359381/2009 [online]. http://www.ema.europa.eu/docs/en_GB/document_ library/Report/2010/01/WC500051739.pdf [27 Sep 2012]. World Health Organization. Causality assessment of an adverse event following immunization (AEFI)—user manual for the revised WHO classification, WHO/ HIS/EMP/QSS.March 2013. Arimone Y. A new method for assessing drug causation provided agreement with expert’s judgement. J Clin Epidemiol 2006; 59: 308–314. Swaen G. A weight of evidence approach to causal inference. J Clin Epidemiol 2009; 62: 270–277. Sauzet O. Illustration of the weibull shape parameter signal detection tool using electronic healthcare record data. Drug Saf. DOI 10.1007/s40264-013-0061-7 Cornelius VR. A signal detection method to detect adverse drug reactions using a parametric time-to-event model in simulated cohort data. Drug Saf 2012; 35(7): 599–610. Norén GN. Temporal pattern discovery for trends and transient effects: its application to patient records. Proceedings of the Fourteenth International Conference on Knowledge Discovery and Data Mining SIGKDD 2008, 963–971. Norén GN. Temporal pattern discovery in longitudinal electronic patient records. Data Min Knowl Discovery 2010; 20(3): 361–387. Van Holle L. The upper bound to the Relative Reporting Ratio—a measure of the impact of the violation of hidden assumptions underlying some disproportionality methods used in signal detection. Pharmacoepidemiol Drug Saf 2014. DOI:10.1002/pds.3556. Stang PE. Advancing the science for active surveillance: rationale and design for the Observational Medical Outcomes Partnership. Ann Intern Med 2010; 153: 600–606.
Pharmacoepidemiology and Drug Safety, (2014); 23: 849–858 DOI: 10.1002/pds
