The performance of adverse effects search ... - Wiley Online Library

DOI: 10.1111/j.1471-1842.2012.00980.x

The performance of adverse effects search filters in MEDLINE and EMBASE Su Golder* & Yoon K. Loke† *Centre for Reviews and Dissemination (CRD), University of York, York, UK and †School of Medicine, Health Policy and Practice, University of East Anglia, Norwich, UK

Abstract Background: Search filters can potentially improve the efficiency of searches involving electronic databases such as MEDLINE and EMBASE. Although search filters have been developed for identifying records that contain adverse effects data, little is known about the sensitivity of such filters. Objectives: This study measured the sensitivity of using available adverse effects filters to retrieve papers with adverse effects data. Methods: A total of 233 included studies from 26 systematic reviews of adverse effects were used for analysis. Search filters from MEDLINE and EMBASE were tested for their sensitivity in retrieving the records included in these reviews. In addition, the sensitivity of each individual search term used in at least one search filter was measured. Results: Subheadings proved the most useful search terms in both MEDLINE and EMBASE. No indexing terms in MEDLINE achieved over 12% sensitivity. The sensitivity of published search filters varied in MEDLINE from 3% to 93% and in EMBASE from 57% to 97%. Whether this level of sensitivity is acceptable will be dependent on the purpose of the search. Conclusions: Although no adverse effects search filter captured all the relevant records, high sensitivity could be achieved. Search filters may therefore be useful in retrieving adverse effects data. Keywords: adverse effects, information storage and retrieval,

MEDLINE,

safety, toxicity.

Key Messages Implications for Practice • A more sensitive search for adverse effects data can be constructed with greater ease in EMBASE than in MEDLINE. • Subheadings are useful in MEDLINE and EMBASE when searching for adverse effects. • No indexing terms were particularly useful in MEDLINE. • Adverse effects search filters can be used for focused searches of MEDLINE and EMBASE with reasonable sensitivity. Implications for Policy • The performance of adverse effects search filters in terms of precision should be measured in a wide range of prospectively conducted studies in order to determine the effectiveness of such filters in practice. • The indexing of records with adverse effects data needs improvement. Introduction Correspondence: Su Golder, Research Fellow in Health Services Research, Centre for Reviews and Dissemination (CRD), University of York, York YO10 5DD, UK. E-mail: [email protected]

Search filters (predefined combinations of search terms) have been developed to capture records for numerous different study designs [such as randomised

ª 2012 The authors. Health Information and Libraries Journal ª 2012 Health Libraries Group Health Information and Libraries Journal, 29, pp.141–151

141

142

Adverse effects search filters, Su Golder & Yoon K. Loke

controlled trials (RCTs) and systematic reviews] and topic areas (such as by age category or disease). Search filters are useful for searchers as they can improve the effectiveness of searches and reduce the time needed to create search strings. Filters can be developed via expert opinion or through objective research methods, or a combination of both. However, it can be difficult to create search filters that achieve an acceptable sensitivity and precision. One area that is particularly problematic is the development of adverse effects filters. An adverse effect is defined as ‘A harmful or undesirable outcome that occurs during or after the use of a drug or intervention for which there is at least a reasonable possibility of a causal relation’.1 Adverse effects are an important consideration for patients, clinicians and other decision makers (such as NICE) to make informed balanced decisions, on whether the potential benefits of an intervention outweigh its potential harm.2,3 Identifying appropriate information on adverse effects, however, can be challenging, and attempts have been made to ease this process with the creation of adverse effects search filters. A key difficulty in creating adverse effects search filters stems from the absence of adverse effects terms in the title, abstract and indexing of relevant articles. In 2001, Derry et al.4 evaluated trials that reported adverse effects data and found that about 23% of such trials had no adverse effects terms (either generic, such as adverse effect or side effect, or specific, such as headache or rash) in the title, abstract or indexing of records in MEDLINE or EMBASE. Despite a lack of reported adverse effects terms, a number of adverse effects filters have still been proposed for MEDLINE5–11 and 7,8 EMBASE. However, the sensitivity of using such filters is of concern given that nearly a quarter of relevant papers do not have any adverse effect– related terms. This lack of confidence in adverse effects search filters has led to current guidance that emphasises the use of non-specific searches (without relying on adverse effects search filters) as well as the need to check full-text versions of retrieved articles.12 Unfortunately, implementation of such guidance in systematic reviews is onerous and time-consuming compared to running more specific adverse effect–based searches.

In this study, we planned to measure the performance, in terms of sensitivity, of published search filters for MEDLINE and EMBASE as well as individual search terms in identifying papers with adverse effects data. Methods Selection of studies containing adverse effects data A collection of papers that report data on the frequency of adverse effects was sought from the studies of systematic reviews of adverse effects. The use of included papers from systematic reviews has been shown to be an effective way of identifying a standard set of records for use in studies evaluating search strategies.13 The included reviews may have evaluated adverse effects in different ways, for instance, singling out specific named adverse clinical outcomes of interest (such as fractures or myocardial infarction), or they may have taken a broader look at the general safety or tolerability profile without limiting their review to particular adverse outcomes. Identification of systematic reviews of adverse effects Systematic reviews of adverse effects were identified through browsing the Database of Abstracts of Reviews of Effects (DARE) (http://www.crd. york.ac.uk/crdweb/). A systematic review was considered eligible for inclusion if it was published between 2007 and 2010 and d Adverse drug effects were the primary outcome. d Adverse effects search terms or specified ⁄ named adverse effects had not been used by the review authors. This enabled us to build an unselected cohort, that is, where relevant articles had not already been chosen because of the presence of specific adverse effects terms. Typically, such reviews would have relied on search terms for the population ⁄ condition and intervention only, such as type II diabetes and rosiglitazone. d The search included either hand searching or reference checking in addition to database searches.



Selection of relevant primary studies with adverse effects data The included references in each systematic review were checked for papers published in English after the year 2000, so that a contemporaneous study cohort could be obtained. Non-English language papers were excluded as we were concerned about obtaining valid matches for adverse effects terms in different languages. Full-text articles were checked to confirm the presence of adverse effects data that had been used in the systematic review. The papers were then de-duplicated, in order to remove copies of papers that had been included in more than one systematic review.

Analysis Availability in selected databases. The first stage of the analysis was to check whether each paper was listed and indexed in MEDLINE or EMBASE. In order to ascertain this, several iterations using author names and words from the title were used.

Published search filters. We then checked whether the papers available on MEDLINE and EMBASE would have been identifiable using a published search filter. Each filter was tested in turn and the sensitivity of using each filter recorded. Owing to the logistical constraints of repeating search strategies, it was not possible to measure the precision of the search filters. However, it could be safely assumed that precision will be higher with a search filter as opposed to without a filter, as fewer records would be retrieved with a filter. The MEDLINE search filters tested were Badgett et al.,5,6 Golder et al.,7 BMJ Clinical Evidence,8 and Buckingham et al.,9 and the EMBASE filters; Golder et al.,7 and BMJ Clinical Evidence 2006 (Box 1).8 The PubMed filters by Wieland and Dickersin10,11 were not tested as these filters were designed specifically for a review on oral contraceptives and breast cancer (Box 1). The search filters by Badgett et al.5,6 and Golder et al.7 do not contain any specific named adverse effects terms and could therefore be tested on all the included papers. The other filters (including a

Box 1 Published adverse effects filters Search Strategies excluding specified named adverse effects terms MEDLINE

Badgett5,6 (ae OR co OR po OR de).fs OR CASE REPORT ⁄ and HUMAN ⁄ (no need to preselect specific ADRS) Golder7 Most sensitive search strategy excluding use of specified named adverse effects (ae OR co OR de).fs OR (safe OR safety OR side effect* OR undesirable effect* OR treatment emergent OR tolerability OR toxicity OR adrs OR (adverse adj2 (effect OR effects OR reaction OR reactions OR event OR events OR outcome OR outcomes)).ti,ab EMBASE

Golder7 Most sensitive search strategy excluding use of specified named adverse effects DRUG ⁄ ae, to OR (safe OR safety OR side effect* OR undesirable effect* OR treatment emergent OR tolerability OR toxicity OR adrs OR (adverse adj2 (effect OR effects OR reaction OR reactions OR event OR events OR outcome OR outcomes))).ti,ab Search Strategies including specified named adverse effects terms MEDLINE

BMJ Clinical Evidence8 Specified named adverse effect AND (ae OR to OR po OR co).fs. OR (safe OR safety).ti,ab. OR side effect$.ti,ab. OR ((adverse OR undesirable OR harm$ OR serious OR toxic) adj3 (effect$ OR reaction$ OR event$ OR outcome$)).ti,ab. OR exp product surveillance, postmarketing ⁄ OR exp adverse drug reaction reporting systems ⁄ OR exp clinical trials, phase iv ⁄ OR exp poisoning ⁄ OR exp substance-related disorders ⁄ OR exp drug toxicity ⁄ OR exp abnormalities, drug induced ⁄ OR exp drug monitoring ⁄ OR exp drug hypersensitivity ⁄ OR (toxicity OR complication$ OR noxious OR tolerability).ti,ab. OR exp Postoperative Complications ⁄ OR exp Intraoperative Complications ⁄ Buckingham9 Step 1. Source of harm with possible subheadings ª 2012 The authors. Health Information and Libraries Journal ª 2012 Health Libraries Group Health Information and Libraries Journal, 29, pp.141–151

143

144

Adverse effects search filters, Su Golder & Yoon K. Loke Box 1 Continued. Adverse Effects (AE), Mortality (MO) (not always an option), Contraindications (CT) Poisoning (PO), Toxicity (TO) Step 2. Disease or disorder ⁄ outcome with possible subheadings Abnormalities (AB), Chemically Induced (CI), Epidemiology (EP), Etiology (ET) Mortality (MO), Transmission (TM) Step 3. Combine Previous Searches with a Quality Filter Combine the first two search statements, using the boolean operator ‘and’ and combine the final subject statement with the quick filter (hedge) below Case control studies ⁄ OR cohort studies ⁄ OR risk ⁄ Golder7 Most sensitive search strategy Specified named adverse effects ⁄ ci OR (ae OR co OR de).fs OR (safe OR safety OR side effect* OR undesirable effect* OR treatment emergent OR tolerability OR toxicity OR adrs OR (adverse adj2 (effect OR effects OR reaction OR reactions OR event OR events OR outcome OR outcomes)).af Wieland10,11 Exploding MeSH term search 1966:1995 [dp] AND ‘human’ [MESH] AND journal article [pt] AND breast neoplasms [mh] AND contraceptives, oral [mh] AND (risk [mh] OR follow-up studies [mh] OR case-control studies [mh]) Maximise Precision: MeSH term search with major topics and subheadings 1966:1995 [dp] AND ‘human’ [MESH] AND journal article [pt] AND ‘breast neoplasms’ [majr:noexp] AND (contraceptives, oral [mh:noexp] OR contraceptives, oral ⁄ pharmacology [mh] OR contraceptives, oral ⁄ therapeutic use [mh] OR estrogens ⁄ therapeutic use [mh] OR contraceptives, oral ⁄ adverse effects [mh]) AND (risk [mh:noexp] OR risk factors [mh:noexp] OR follow-up studies [mh:noexp] OR odds ratio [mh:noexp]) MeSH term search without study methodology terms 1966:1995 [dp] AND ‘human’ [MESH] AND journal article [pt] AND ‘breast neoplasms’ [majr:noexp] AND (contraceptives, oral [mh:noexp] OR contraceptives, oral ⁄ pharmacology [mh] OR estrogens ⁄ therapeutic use [mh] OR contraceptives, oral ⁄ therapeutic use [mh] OR contraceptives, oral ⁄ adverse effects [mh]) Maximise Sensitivity: MeSH term search without intervention terms 1966:1995 [dp] AND ‘human’ [MESH] AND journal article [pt] AND ‘breast neoplasms’ [majr:noexp] AND (risk [mh:noexp] OR risk factors [mh:noexp] OR follow-up studies [mh:noexp] OR odds ratio [mh:noexp]) Text word search with automatic term mapping 1966:1995 [dp] AND ‘human’ [MESH] AND journal article [pt] AND breast cancer AND (oral contraceptive OR oral contraceptives OR estrogen OR estrogens OR hormones OR hormonal) AND (risk OR follow-up OR epidemiologic) Text word search with truncation and double quotes 1966:1995 [dp] AND ‘human’ [MESH] AND journal article [pt] AND ‘breast cancer’ AND (oral contraceptive* OR ‘estrogen’ OR ‘hormones’ OR ‘hormonal’) AND (‘risk’ OR ‘epidemiologic’) Text word search without study methodology text words 1966:1995 [dp] AND ‘human’ [MESH] AND journal article [pt] AND ‘breast cancer’ AND (oral contraceptive* OR ‘estrogen’ OR ‘hormones’ OR ‘hormonal’) Maximise Sensitivity: Text word search without intervention text words 1966:1995 [dp] AND ‘human’ [MESH] AND journal article [pt] AND ‘breast cancer’ AND (‘risk’ OR epidemiolog*) EMBASE

BMJ Clinical Evidence8 Specified named adverse effect AND (ae OR si OR to OR co).fs. OR (safe OR safety).ti,ab. OR side effect$.ti,ab. OR ((adverse OR undesirable OR harm$ OR serious OR toxic) adj3 (effect$ OR reaction$ OR event$ OR outcome$)).ti,ab. OR exp adverse drug reaction ⁄ OR exp drug toxicity ⁄ OR exp intoxication ⁄ OR exp drug safety ⁄ OR exp drug monitoring ⁄ OR exp drug hypersensitivity ⁄ OR exp postmarketing surveillance ⁄ OR exp drug surveillance program ⁄ OR exp phase iv clinical trial ⁄ OR (toxicity OR complication$ OR noxious OR tolerability).ti,ab. OR exp postoperative complication ⁄ OR exp Peroperative Complication ⁄ Golder7 Most sensitive search strategy Specified named adverse effects OR (safe OR safety OR side effect* OR undesirable effect* OR treatment emergent OR tolerability OR toxicity OR adrs OR (adverse adj2 (effect OR effects OR reaction OR reactions OR event OR events OR outcome OR outcomes))).ti,ab



second filter by Golder et al.7) include specific named adverse effects terms and were therefore only tested on those included papers identified from a systematic review of a named adverse effect of interest, and not those reviews that aimed to carry out a generalised broad evaluation of safety. In addition to the search combinations proposed in the published search filters, each term included in at least one search filter was tested individually and the sensitivity measured (this included terms from Wieland and Dickersin10,11). This was carried out to ascertain whether specific terms included in the search filters are particularly useful in retrieving papers containing adverse effects data. Results Systematic reviews Twenty-six systematic reviews met the inclusion criteria (Table 2).14–39 Half were concerned with the safety profile for an intervention,14,15,20,23–28,31–33,36 whereas the other half were limited to a named ⁄ specific adverse effect with an intervention.16–19,21,22,29,30,34,35,37–39

Availability of records on

MEDLINE

and

EMBASE

A total of 231 references were indexed on MED119 papers were from reviews on a specific named adverse effect and 117 from safety profile reviews (five references were included in both types of review). A total of 222 references were indexed on EMBASE (two of which were not indexed on MEDLINE), 114 papers from reviews on a specific named adverse effect and 113 from safety profile reviews (five references were included in both types of review).

LINE;

Published search filters (excluding specific named adverse effects terms) Search filters that are based on generic adverse effects terms (such as ‘adverse effect’ or ‘side effect’ or ‘tolerability’ or ‘toxicity’) can potentially be used in any systematic review and were tested on all the included studies. If the Badgett et al.5,6 and Golder et al.7 search filters had been applied in MEDLINE, 28% (65 ⁄ 231) and 13% (29 ⁄ 231) of papers would have been missed and 12% (26 ⁄ 222) of papers would have been missed if the Golder et al.7 search filter had been applied in EMBASE (Table 1).

Primary studies A total of 474 papers were initially found within the systematic reviews. Subsequently, 241 papers were excluded from the analysis; 140 were published before 2001, eight were published in a non-English language, 35 were not indexed on either MEDLINE or EMBASE, 25 were duplicate papers (contained in more than one systematic review), and 33 papers did not contain adverse effects data in the full text relevant to the systematic review in which they were cited. Those references not identified in any of the databases were 15 company reports, eight conference papers, two Food and Drug Administration (FDA) reports and one book. A total of 233 papers were eligible for use in the analysis. The majority of the references were RCTs (90%, 209 ⁄ 233); however, there were also 12 case series, five chart reviews, three case reports, two cohort studies, one non-RCT study and one uncontrolled study.

Published search filters (including specific named adverse effects terms) Those filters dictating the use of specific named adverse effects terms were tested only with the included papers from systematic reviews of specific named adverse effects. Here, 77% (92 ⁄ 119), 93% (111 ⁄ 119), 97% (116 ⁄ 119) and 7% (8 ⁄ 119) of papers would have been missed if the BMJ Clinical Evidence,8 the Buckingham et al.9 variants and Golder et al.7 search filters had been applied in MEDLINE, and 43% (49 ⁄ 114) and 4% (5 ⁄ 114) of papers would have been missed if the BMJ Clinical Evidence8 and Golder et al.7 search filter had been applied in EMBASE (Table 1). The sensitivity of the BMJ Clinical Evidence8 filter would have been improved if the specific named adverse effects terms had been ORed instead of ANDed with the generic adverse effects terms. A total of 109 papers would have been retrieved in MEDLINE with a sensitivity of 92%


145

146

Adverse effects search filters, Su Golder & Yoon K. Loke Table 1 Sensitivity of published strategies for adverse effects

Search strategies excluding specified named adverse effects terms MEDLINE (n = 231) Badgett5,6 Golder7 EMBASE (n = 222) Golder7 Search strategies including specified named adverse effects terms MEDLINE (n = 119) BMJ Clinical Evidence8 Buckingham9 Without the quick filter (hedge) Buckingham9 With the quick filter (hedge) Golder7 EMBASE (n = 114) BMJ clinical evidence8 Golder7

(109 ⁄ 119) and 109 in 96% (109 ⁄ 114).

MEDLINE

and

EMBASE

search

Number of relevant records

Sensitivity (%)

166 202

72 87

195

88

events’ (29%, 67 ⁄ 231) and ‘risk’ (28%, 64 ⁄ 231) in the title or abstract, and the subheadings ‘drug effects (de)’ (27%, 62 ⁄ 231) and ‘complications (co)’ (18%, 41 ⁄ 231). None of the indexing terms retrieved a large proportion of the relevant records. In EMBASE, the highest number of records from EMBASE was retrieved with the floating subheadings, ‘adverse drug reaction’ (ae) (83%, 185 ⁄ 222) and ‘side effect’ (si) (83%, 185 ⁄ 222), this was followed by the EMTREE term exp drug safety ⁄ (38%, 85 ⁄ 222) and the terms ‘adverse adj3 event$’ (32%, 71 ⁄ 222), ‘safety’ (28%, 63 ⁄ 222), ‘adverse adj2 events’ (28%, 63 ⁄ 222) and ‘risk’ (27%, 61 ⁄ 222). Discussion

27 8

23 7

3

3

111

93

65 109

57 96

EMBASE

with a sensitivity of

Different study designs Of the 24 non-RCT studies in MEDLINE, all were retrieved by the BMJ Clinical Evidence8, and the Golder et al.7 search strategies and the search strategy by Badgett et al.5,6 retrieved all but three chart reviews and one non-RCT. Twenty three non-RCT studies were available in EMBASE, and all 23 were retrieved by the BMJ Clinical Evidence8 and the Golder et al.7 search strategies. Individual search terms Table 2 shows the number of relevant records retrieved using each individual generic search term contained in at least one published search filter. In MEDLINE, the most commonly used terms were the subheadings ‘therapeutic use (tu)’ (77%, 177 ⁄ 231) and ‘adverse effects (ae)’ (51%, 117 ⁄ 231) followed by the terms ‘adverse adj3 event$’ (32%, 75 ⁄ 231), ‘safety’ (31%, 71 ⁄ 231), ‘adverse adj2

The search filters by Badgett et al.5,6 and Golder et al.7 achieved fairly high sensitivity, and the filter by BMJ Clinical Evidence8 would also have achieved high sensitivity had the specific named adverse effects been ORed instead of ANDed with the generic adverse effects terms. Whether this level of sensitivity is acceptable for a systematic review search will depend on the topic under evaluation, resources available and anticipated gain in precision. For example, if the search filters reduced the numbers of records needed to sift from an unmanageable set of tens of thousands of records, these search filters may be worth using. High precision is often at the loss of sensitivity though. For instance, the Buckingham filters9 aim to achieve high precision and it is therefore not surprising that these filters fared poorly with respect to sensitivity. The inevitable trade-off between sensitivity and precision may or may not be problematic depending on the particular review, for example, in situations where the search filters reduce the numbers of records needed to sift by only a handful, then any loss in sensitivity may not be acceptable. Evaluation of adverse effects may involve the inclusion of a range of different study designs, such as non-randomised pharmacoepidemiological studies. Hence, the ability to accurately identify relevant observational studies would be a potentially useful feature of a search filter. Although based on a smaller sample, the results here indicate that the BMJ Clinical Evidence8 and the Golder et al.7 search strategies may be particularly useful


Adverse effects search filters, Su Golder & Yoon K. Loke Table 2 Performance of individual search terms from published search filters in

MEDLINE

Floating subheadings (n = 231) Adverse effects (ae) Complications (co) Drug effects (de) Poisoning (po) Toxicity (to) Subheadings attached to intervention (n = 231) Adverse effects (ae) Contraindications (ct) Mortality (mo) Pharmacology (pd) Poisoning (po) Therapeutic use (tu) Toxicity (to) Subheadings attached to disease or disorder ⁄ outcome Abnormalities (ab) Chemically induced (ci) Epidemiology (ep) Etiology (et) Mortality (mo) Transmission (tm) MeSH terms Exp abnormalities, drug induced ⁄ Exp adverse drug reaction reporting systems ⁄ Case control studies ⁄ Exp case control studies ⁄ Case report ⁄ Exp clinical trials, phase iv as topic ⁄ Cohort studies ⁄ Exp drug hypersensitivity ⁄ Exp drug monitoring ⁄ Exp drug toxicity ⁄ Exp follow-up studies ⁄ Human ⁄ Exp Intraoperative Complications ⁄ odds ratio ⁄ Exp poisoning ⁄ Exp Postoperative Complications ⁄ Exp product surveillance, postmarketing ⁄ Risk ⁄ Exp Risk ⁄ Risk factors ⁄ Exp Substance-related disorders ⁄ Terms in the title or abstract adrs adverse adj2 effect adverse adj2 effects adverse adj3 effect$ adverse adj2 event adverse adj2 events

Number of Relevant Records (n = 231)

117 (51%) 41 (18%) 62 (27%) 0 0 117 (51%) 0 0 22 (10%) 0 177 (77%) 0 (n = 119) 0 7 (%) 2 (1%) 0 3 (%) 0 0 0 1 (0.5%) 8 (3%) 0 0 3 (1%) 2 (1%) 0 3 (1%) 24 (10%) 231 (100%) 3 (1%) 1 (0.5%) 4 (2%) 1 (0.5%) 0 6 (3%) 28 (12%) 19 (8%) 6 (3%) 0 4 (2%) 17 (%) 19 (8%) 14 (6%) 67 (29%)

MEDLINE

and

EMBASE

EMBASE

Number of Relevant Records (n = 222)

Floating subheadings Adverse drug reaction (ae) Complication (co) Drug toxicity (to) Side effect (si)

185 (83%) 24 (11%) 2 (1%) 185 (83%)

Subheadings attached to intervention Adverse drug reaction (ae) Drug toxicity (to)

185 (83%) 2 (1%)

EMTREE terms exp adverse drug reaction ⁄ exp drug hypersensitivity ⁄ exp drug monitoring ⁄ exp drug safety ⁄ exp drug surveillance program ⁄ exp drug toxicity ⁄ exp intoxication ⁄

Terms in the title or abstract adrs adverse adj2 effect adverse adj2 effects adverse adj3 effect$ adverse adj2 event adverse adj2 events


43 (19%) 4 (2%) 0 85 (38%) 0 0 0

0 4 (2%) 15 (7%) 19 (9%) 13 (6%) 63 (28%)

147

148

Adverse effects search filters, Su Golder & Yoon K. Loke Table 2 Continued.

MEDLINE

adverse adj3 event$ adverse adj2 outcome adverse adj2 outcomes adverse adj3 outcome$ adverse adj2 reaction adverse adj2 reactions adverse adj3 reaction$ complication$ epidemiolog* epidemiologic follow-up harm$ adj3 effect$ harm$ adj3 event$ harm$ adj3 outcome$ harm$ adj3 reaction$ noxious risk safe safety serious adj3 effect$ serious adj3 event$ serious adj3 outcome$ serious adj3 reaction$ side effect$ tolerability toxic adj3 effect$ toxic adj3 event$ toxic adj3 outcome$ toxic adj3 reaction$ toxicity treatment emergent undesirable effect* undesirable adj3 effect$ undesirable adj3 event$ undesirable adj3 outcome$ undesirable adj3 reaction$

Number of Relevant Records (n = 231) 75 (32%) 0 0 0 0 3 (1%) 3 (1%) 7 (3%) 0 0 29 (13%) 0 0 0 0 0 64 (27%) 17 (7%) 71 (31%) 1 (0.5%) 15 (6%) 0 2 (1%) 22 (10%) 23 (10%) 2 (1%) 0 0 0 11 (5%) 0 0 0 0 0 0

in retrieving non-randomised studies. This may be because many observational studies are focused on adverse effects as primary rather than secondary outcomes and therefore easier to retrieve. To narrow things down further, our analysis of the utility of particular search terms can help in the design of new filters or in guiding the modification of existing ones for improved performance. Here, we found the most useful individual search terms in both MEDLINE and EMBASE were adverse effects subheadings. These tended to retrieve more

EMBASE

adverse adj3 event$ adverse adj2 outcome adverse adj2 outcomes adverse adj3 outcome$ adverse adj2 reaction adverse adj2 reactions adverse adj3 reaction$ complication$ epidemiolog* epidemiologic follow-up harm$ adj3 effect$ harm$ adj3 event$ harm$ adj3 outcome$ harm$ adj3 reaction$ noxious risk safe safety serious adj3 effect$ serious adj3 event$ serious adj3 outcome$ serious adj3 reaction$ side effect$ tolerability toxic adj3 effect$ toxic adj3 event$ toxic adj3 outcome$ toxic adj3 reaction$ toxicity treatment emergent undesirable effect* undesirable adj3 effect$ undesirable adj3 event$ undesirable adj3 outcome$ undesirable adj3 reaction$

Number of Relevant Records (n = 222) 71 (32%) 0 0 0 0 3 (1%) 3 (1%) 7 (3%) 0 0 28 (13%) 0 0 0 0 0 61 (27%) 17 (8%) 63 (28%) 1 (0.5%) 12 (5%) 0 2 (1%) 23 (10%) 24 (11%) 2 (1%) 0 0 0 11 (5%) 0 0 0 0 0 0

relevant records than either text words or indexing terms. In the common scenario where a customised search strategy for a particular topic needs to be built from scratch, it would be sensible to design the search with a greater emphasis on the inclusion of relevant subheadings (based on examples in Table 2), rather than rely on picking up specific terms in the title or abstract. Subheadings are likely to be more useful as there is a dearth of appropriate indexing terms available. In addition, adverse effects terms are less likely to appear in



the title and abstract, and when they do, there is a lack of consistency in the terms used. Limitations The main limitation of this research is that the precision of search terms and search filters could not be measured. Although this study indicates which search terms are most effective in retrieving papers with adverse effects data, these terms may still retrieve a high proportion of irrelevant records (but less so than similar searches with no adverse effects terms at all). In developing any search strategy, it is important to obtain a balance between sensitivity and precision. The most sensitive searches may also be those with the lowest precision. Caution should therefore be applied when using the results of this study, especially as some of the search terms appeared to have only a vague connection to adverse effects (such as ‘therapeutic use’, ‘pharmacology’ ‘follow-up’ or ‘risk’). Although systematic review searches aim to be as comprehensive as possible, such a search may yield several thousand articles and only a small fraction of them will be relevant. Search results should aim to be manageable in terms of the feasibility of screening all the potentially relevant studies, particularly as previous recommendations have suggested that checking full-text articles may be necessary. Conclusions In our evaluation of search terms for adverse effects data, subheadings (either free floating or with subject heading) seem to be the most useful means of identifying relevant papers in both MEDLINE and EMBASE. Few indexing terms exist for adverse effects, and the sensitivity of those that do exist is low, particularly in MEDLINE. While some free text terms for adverse effects in the title and abstract may be useful, we recommend that free text terms should only be applied concomitantly with other terms such as subheadings. The sensitivity of adverse effects search filters varied considerably, and although high sensitivity could be achieved in both MEDLINE and EMBASE, 100% sensitivity was not achievable. Further research that also measures the precision of the search terms and filters is required to

establish the full value of using adverse effects terms in search strategies. In the meantime, improvements in the allocation of subheadings to bibliographic records could ease the workload of the systematic reviewer. Acknowledgements We would like to thank Professor Lesley Stewart, CRD for comments on an earlier draft. Source of Funding This research was undertaken by Su Golder as part of a Medical Research Council (MRC) fellowship. The views expressed in this presentation are those of the authors and not necessarily those of the MRC. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. Conflict of Interest No conflicts of interests are declared by the authors. References 1 Chou, R., Aronson, N., Atkins, D., Ismaila, A. S., Santaguida, P., Smith, D. H., Whitlock, E., Wilt, T. J. & Moher, D. AHRQ Series Paper 4: Assessing harms when comparing medical interventions: AHRQ and the Effective Health-Care Program. Journal of Clinical Epidemiology 2010, 63, 502–512. 2 Keech, A. C., Wonders, S. M., Cook, D. I. & Gebski, V. J. Balancing the outcomes: reporting adverse events. Medical Journal of Australia 2004, 181, 215–218. 3 Strom, B. L. What is pharmacoepidemiology? In: Strom, B. L. (ed.). Pharmacoepidemiology. Chichester, PA: John Wiley, 2006: 3–11. 4 Derry, S., Loke, Y. K. & Aronson, J. K. Incomplete evidence: the inadequacy of databases in tracing published adverse drug reactions in clinical trials. BMC Medical Research Methodology 2001, 1, 7. 5 Badgett, R., Chiquette, E., Anagnostelis, B. & Mulrow, C. Locating Reports of Serious Adverse Drug Reactions (PowerPoint Presentation), 1999. Accessible at: http:// medinformatics.uthscsa.edu/#FILTERS, accessed 7 January 2005. 6 Badgett, R., Chiquette, E., Anagnostelis, B. & Mulrow, C. Locating reports of serious adverse drug reactions. 7th Annual Cochrane Colloquium Abstracts, October 1999.


149

150


7

8

9

10

11

12

13

14

15

16

17

18 19

Rome, Italy, 1999. Accessible at: http://www.imbi.unifreiburg.de/OJS/cca/index.php?journal=cca&page=article& op=view&path[]=4565, accessed 15 February 2012. Golder, S., McIntosh, H. M., Duffy, S. & Glanville, J. Developing efficient search strategies to identify reports of adverse effects in MEDLINE and EMBASE. Health Information and Libraries Journal 2006, 23, 3–12. BMJ Clinical Evidence. Search Filters. BMJ, 2006. Accessible at: http://clinicalevidence.bmj.com/x/set/static/ ebm/learn/665076.html, accessed 14 February 2012. Buckingham, J., Fisher, B. & Saunders, D. Ovid Medline ⁄ Pubmed Search Strategies for Articles on Causation ⁄ Harm ⁄ Etiology. Alberta: Evidence Based Medicine Tool Kit, 2005. Accessible at: http://www.ebm. med.ualberta.ca/HarmSearch.html, accessed 13 May 2005. Wieland, S. & Dickersin, K. Selective exposure reporting and Medline indexing limited the search sensitivity for observational studies of the adverse effects of oral contraceptives. Journal of Clinical Epidemiology 2005, 58, 560–567. Wieland, S. & Dickersin, K. Erratum: selective exposure reporting and Medline indexing limited the search sensitivity for observational studies of the adverse effects of oral contraceptives (Journal of Clinical Epidemiology (June 2005) 58:6 (560–567)). Journal of Clinical Epidemiology 2005, 58, 1077. Higgins, J. P. T. & Green, S. (eds). Cochrane Handbook for Systematic Reviews of Interventions 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Accessible at: http://www.cochrane-handbook.org. Sampson, M., Zhang, L., Morrison, A., Barrowman, N. J., Clifford, T. J., Platt, R. W., Klassen, T. P. & Moher, D. An alternative to the hand searching gold standard: validating methodological search filters using relative recall. BMC Medical Research Methodology 2006, 6, 33. Agbabiaka, T., Pittler, M., Wider, B. & Ernst, E. Serenoa repens (saw palmetto): a systematic review of adverse events. Drug Safety 2009, 32, 637–647. Albavera-Herna´ndez, C., Rodrı´guez, J. & Idrovo, A. Safety of botulinum toxin type A among children with spasticity secondary to cerebral palsy: a systematic review of randomized clinical trials. Clinical Rehabilitation 2009, 23, 394–407. Alghamdi, A. A., Moussa, F. & Fremes, S. E. Does the use of preoperative aspirin increase the risk of bleeding in patients undergoing coronary artery bypass grafting surgery? Systematic review and meta-analysis. Journal of Cardiac Surgery 2007, 22, 247–256. Berlie, H., Kalus, J. & Jaber, L. Thiazolidinediones and the risk of edema: a meta-analysis. Diabetes Research and Clinical Practice 2007, 76, 279–289. Bonovas, S. & Sitaras, N. Does pravastatin promote cancer in elderly patients? A meta-analysis. CMAJ 2007, 176, 649–654. Cao, Y., Liu, L., Liao, C., Tan, A. & Gao, F. Meta-analysis of incidence and risk of hypokalemia with cetuximab-based therapy for advanced cancer. Cancer Chemotherapy and Pharmacology 2010, 66, 37–42.

20 Chavez-Tapia, N., Tellez-Avila, F., Bedogni, G., Croce`, L., Masutti, F. & Tiribelli, C. Systematic review and meta-analysis on the adverse events of rimonabant treatment: considerations for its potential use in hepatology. BMC Gastroenterology 2009, 9, 75. 21 Chen, L. & Ashcroft, D. Risk of myocardial infarction associated with selective COX-2 inhibitors: meta-analysis of randomised controlled trials. Pharmacoepidemiology and Drug Safety 2007, 16, 762–772. 22 Correll, C. Weight gain and metabolic effects of mood stabilizers and antipsychotics in pediatric bipolar disorder: a systematic review and pooled analysis of short-term trials. Journal of the American Academy of Child and Adolescent Psychiatry 2007, 46, 687–700. 23 Dugoua, J., Machado, M., Zhu, X., Chen, X., Koren, G. & Einarson, T. Probiotic safety in pregnancy: a systematic review and meta-analysis of randomized controlled trials of Lactobacillus, Bifidobacterium, and Saccharomyces spp. Journal of Obstetrics and Gynaecology, Canada 2009, 31, 542–552. 24 Ford, A., Malfertheiner, P., Giguere, M., Santana, J., Khan, M. & Moayyedi, P. Adverse events with bismuth salts for Helicobacter pylori eradication: systematic review and meta-analysis. World Journal of Gastroenterology 2008, 14, 7361–7370. 25 Gehling, M. & Tryba, M. Riks and side-effects of intrathecal morphine combined with spinal anaesthesia: a meta-analysis. Anaesthesia 2009, 64, 643–651. 26 Johansson, K., Neovius, K., DeSantis, S., Ro¨ssner, S. & Neovius, M. Discontinuation due to adverse events in randomized trials of orlistat, sibutramine and rimonabant: a meta-analysis. Obesity Reviews 2009, 10, 564–575. 27 Lakhdar, R., Al-Mallah, M. & Lanfear, D. Safety and tolerability of angiotensin-converting enzyme inhibitor versus the combination of angiotensin-converting enzyme inhibitor and angiotensin receptor blocker in patients with left ventricular dysfunction: a systematic review and meta-analysis of randomized controlled trials. Journal of Cardiac Failure 2008, 14, 181–188. 28 Luykx, J., Mason, M., Ferrari, M. & Carpay, J. Are migraineurs at increased risk of adverse drug responses? A meta-analytic comparison of topiramate-related adverse drug reactions in epilepsy and migraine. Clinical Pharmacology and Therapeutics 2009, 85, 283–288. 29 Mauri, D., Valachis, A., Polyzos, I., Polyzos, N., Kamposioras, K. & Pesce, L. Osteonecrosis of the jaw and use of bisphosphonates in adjuvant breast cancer treatment: a meta-analysis. Breast Cancer Research and Treatment 2009, 116, 433–439. 30 Morris, T., Castrejon, S., Devendra, G. & Gamst, A. No difference in risk for thrombocytopenia during treatment of pulmonary embolism and deep venous thrombosis with either low-molecular-weight heparin or unfractionated heparin: a metaanalysis. Chest 2007, 132, 1131–1139. 31 Phillips, C., Kashani, A., Ko, D., Francis, G. & Krumholz, H. Adverse effects of combination angiotensin II receptor blockers plus angiotensin-converting enzyme inhibitors for



32

33

34

35

left ventricular dysfunction: a quantitative review of data from randomized clinical trials. Archives of Internal Medicine 2007, 167, 1930–1936. Ravindran, V., Rachapalli, S. & Choy, E. Safety of medium- to long-term glucocorticoid therapy in rheumatoid arthritis: a meta-analysis. Rheumatology 2009, 48, 807–811. Rodrigo, G., Nannini, L. & Rodrı´guez-Roisin, R. Safety of long-acting beta-agonists in stable COPD: a systematic review. Chest 2008, 133, 1079–1087. Rodrigo, G., Castro-Rodriguez, J., Nannini, L., Plaza Moral, V. & Schiavi, E. Tiotropium and risk for fatal and nonfatal cardiovascular events in patients with chronic obstructive pulmonary disease: systematic review with meta-analysis. Respiratory Medicine 2009, 103, 1421–1429. Siegel, C., Marden, S., Persing, S., Larson, R. & Sands, B. Risk of lymphoma associated with combination anti-tumor necrosis factor and immunomodulator therapy for the treatment of Crohn’s disease: a meta-analysis. Clinical Gastroenterology and Hepatology 2009, 7, 874–881.

36 Silva, M., Matthews, M. L., Jarvis, C., Nolan, N. M., Belliveau, P., Malloy, M. & Gandhi, P. Meta-analysis of drug-induced adverse events associated with intensive-dose statin therapy. Clinical Therapeutics 2007, 29, 253–260. 37 Singh, S., Loke, Y. K. & Furberg, C. D. Long-term risk of cardiovascular events with rosiglitazone: a meta-analysis. JAMA 2007, 298, 1189–1195. 38 Singh, S., Loke, Y. K. & Furberg, C. D. Inhaled anticholinergics and risk of major adverse cardiovascular events in patients with chronic obstructive pulmonary disease: a systematic review and meta-analysis. JAMA 2008, 300, 1439–1450. 39 Singh, S., Amin, A. V. & Loke, Y. K. Long-term use of inhaled corticosteroids and the risk of pneumonia in chronic obstructive pulmonary disease: a meta-analysis. Archives of Internal Medicine 2009, 169, 219–229. Received 27 July 2011; Accepted 24 January 2012


151