SDMinP: a program to control the family wise error rate using step ...

Bioinformatics Advance Access published May 6, 2005

© The Author (2005). Published by Oxford University Press. All rights reserved. For Permissions, please email: [email protected]

SDMinP: a program to control the family wise error rate using step-down minP adjusted p-values M. Obreiter a , C. Fischer b , J. Chang-Claude a , L. Beckmann a

b

German Cancer Research Center DKFZ, Heidelberg, Germany

Institute of Human Genetics, University of Heidelberg, Germany

Keywords: Family Wise Error Rate, Multiple Testing, Correlated Tests, Step Down minP

Address for correspondence: Dr. Lars Beckmann German Cancer Research Center DKFZ Department of Clinical Epidemiology Im Neuenheimer Feld 280; 69120 Heidelberg, Germany E-mail: [email protected] Phone: ++49 6221 422214; Fax: ++49 6221 422203

1

Downloaded from http://bioinformatics.oxfordjournals.org/ by guest on June 5, 2013

a

Abstract SDMinP is an easy-to-use program for fast calculation of empirical and adjusted p-values for correlated and uncorrelated hypotheses in multiple testing experiments. It is based on the Free Step-Down Resampling Method for controlling the Family Wise Error Rate, originally proposed by (Westfall and Young, 1993), and

the originally required re-sampling effort considerably and made the method computationally feasible. The program is independent of the underlying test statistic and works with provided observed and permutation test statistics.

Availability: http://www.dkfz.de/SDMinP.

1

Introduction

Multiple testing, a scenario in which more than one individual hypothesis are tested simultaneously, requires the control of the multiple type I error rate. One definition of the multiple type I error rate is the Family Wise Error Rate (FWER),

2

Downloaded from http://bioinformatics.oxfordjournals.org/ by guest on June 5, 2013

implements a variation of the efficient algorithm of Ge et al. (2003), who reduced

which denotes the probability of having at least one false significant test result within the set of tested hypotheses. The FWER increases with the number of hypotheses and therefore has to be controlled by adjusting the (raw) p-value of the observed test statistic of each individual hypothesis and obtaining a corresponding adjusted p-value. Common methods are the Bonferroni correction or more refined variations.

testing scenario with correlated hypotheses (e.g., association analysis of multiple markers and gene-gene interaction) leads to conservative results. Thus, multi-step procedures were developed to achieve higher power for correlated tests while controlling the FWER. Westfall and Young (1993) proposed the Free Step-Down Resampling Method, a multi-step procedure for controlling the FWER. This method uses the joint null distribution of p-values, obtained by re-sampling under the global null hypothesis (i.e., under the assumption that all individual null hypotheses are true), to obtain step-down minP adjusted p-values. However, the determination of the joint null distribution of p-values leads to almost infeasible re-sampling effort if the distribution of the test statistics is unknown. In this case, p-values have to be determined empirically by additional re-sampling and permutation steps under

3

Downloaded from http://bioinformatics.oxfordjournals.org/ by guest on June 5, 2013

Applying the Bonferroni correction, which is a single-step procedure, to a multiple

the global null hypothesis. Ge et al. (2003) and Becker and Knapp (2004) improved the original method by reducing the re-sampling effort and made it feasible and attractive. Ge et al. (2003) offer the R-package multtest (available at www.bioconductor.org), which is especially applicable for microarray data analysis. It can only be used with provided standard test statistics. Based on their approaches, we developed SDMinP

from a particular test statistic. In Beckmann et al. (2004) we demonstrated the gain in statistical power when applying this adjustment compared to other methods for controlling the FWER. Ge et al. (2003) reduced the re-sampling complexity, known as ’double-permutation’, considerably and hence lessened the computational effort. The joint null distribution of p-values is calculated on the basis of only one set of permutation test statistics. Furthermore, they proposed an efficient algorithm for the implementation of the method. Becker and Knapp (2004) presented an approach that reduced the re-sampling effort even more. Here, the calculation of the raw p-values of the observed test statistics and the joint null distribution of p-values are based on the same single set of permutation test statistics. They presented two optional formulae for obtaining

4

Downloaded from http://bioinformatics.oxfordjournals.org/ by guest on June 5, 2013

for fast calculation of step-down minP adjusted p-values, which is independent

the global p-value. The first formula determines the global p-value as the smallest adjusted p-value of the individual hypotheses. The second formula is appropriate for relatively small numbers of permutation replicates. It takes the discreteness of the p-value distribution into account and considers also the distribution of the second smallest raw p-values. SDMinP incorporates the suggestions of Ge et al. (2003) and Becker and Knapp

input format (see in section Features: Input Data and Format), empirical raw p-values. The global p-value is determined as well, where both formulae, presented by Becker and Knapp (2004), can be applied. The program is easy-to-use and works with provided observed and permutation test statistics. This makes it appealing to non-standard test statistics, whose distributions are unknown and where p-values have to be estimated empirically by permutation under the global null hypothesis.

2

Features

SDMinP is available for free. A detailed documentation and example files are included in the download package. Program type and Configuration: SDMinP is a command line tool, which is con5

Downloaded from http://bioinformatics.oxfordjournals.org/ by guest on June 5, 2013

(2004). It calculates step-down minP adjusted p-values and, depending on the

trolled by a configuration file with a low number of parameters. It is possible to set calculation and performance parameters and to determine the logging granularity. Algorithms: SDMinP supports two optional approaches for the calculation of unadjusted permutation based raw p-values: Ge et al. (2003) and Becker and Knapp (2004). The formulas differ slightly. For discussion regarding the choice

The global p-value is obtained either by taking the smallest adjusted p-value of the hypotheses or by using the improved formula, presented by Becker and Knapp (2004), which includes the distribution of the second smallest raw p-values. Input Data and Format: Input data are provided via a flat text file. Each line of the input file contains the information of one hypothesis. The required data per hypothesis consists of one unique identifier, the pre-calculated raw p-value (if available, otherwise the placeholder ’NA’ has to be set), the observed test statistic and a user defined number of calculated permutation test statistics. If the placeholder instead of the raw p-value is given, SDMinP calculates the empirical raw pvalue on the basis of the provided test statistics as proposed by Becker and Knapp (2004). The input file can be in the magnitude of megabytes or gigabytes for large

6

Downloaded from http://bioinformatics.oxfordjournals.org/ by guest on June 5, 2013

of formula see Becker and Knapp (2004) and the program documentation.

numbers of hypotheses and permutation test statistics. The performance problem for handling such files has been solved, see Implementation. Statistical Test: The test character, i.e. whether it is left-, right- or two-sided, can be specified in the configuration file. Logging Mechanism: Each single calculation step can be logged by enabling the respective log mechanism. This feature is useful for following up the computa-

mance and gives no easily readable information due to the large amount of data. Results: The results are stored in a result file, consisting of the observed test statistic, the raw- and adjusted p value per hypothesis and the global p-value. Optionally, an additional text file containing the results in an ’R’-readable format (R Development Core Team, 2004) can be created.

3

Implementation

The program is written in Python 2.3.5 (available at www.python.org) and runs in a Windows as well as in an Unix environment. The results of performance tests are presented in table 1. One challenge was to deal with the data input file, which can be considerably large and has to be parsed frequently. We solved this performance problem by splitting the input file into smaller parts, which can be browsed faster. 7

Downloaded from http://bioinformatics.oxfordjournals.org/ by guest on June 5, 2013

tional process of small data sets. For larger data sets, it slows down the perfor-

Acknowledgement We thank Dr. Tim Becker and Dr. Michael Knapp for advice and comments on the statistical methods. This work was supported by a Deutsche Forschungsgemeinschaft grant (CH117/3-1 for LB) and by the Federal Ministry of Education, Science, Research and Technology (NGFN-2 PGE-S19T05 and PGE-S30T09 for MO).

Becker,T.,Knapp,M. (2004) A Powerful Strategy to Account for Multiple Testing in the Context of Haplotype Analysis, Am. J. Hum. Genet., 75, 561-570. Beckmann,L.,Fischer,C.,Chang-Claude,J. (2004) Analysis of multiple error rates in haplotype-based association studies, Abstracts of The American Society of Human Genetics, 54th Annual meeting. 2004. Ge,Y.,Dudoit,S.,Speed,T.P. (2003) Resampling-based Multiple Testing for Microarray Data Analysis, Test, 12, 1-77 R Development Core Team (2004). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL http://www.R-project.org.

8

Downloaded from http://bioinformatics.oxfordjournals.org/ by guest on June 5, 2013

References

Westfall,P.H.,Young,S.S. (1993) Resampling-based multiple testing: examples and methods for P -value adjustment John Wiley & Sons, New York

Downloaded from http://bioinformatics.oxfordjournals.org/ by guest on June 5, 2013

9

10

Downloaded from http://bioinformatics.oxfordjournals.org/ by guest on June 5, 2013

Table 1: Results of the performance test. # Hypotheses # Permutation File Size Time for Time for Test Statistics (GB) Format I Format II 20 100000 0.018 53 sec 72 sec 800 100000 0.75 28 min 38 min 3200 100000 2.9 140 min 166 min 6400 100000 5.8 270 min 308 min The table contains the calculation time for determining step-down minP adjusted p-values for different numbers of hypotheses and permutation test statistics, file sizes and whether the raw p-values of the observed test statistics were provided (Format = I) or had to be calculated (Format = II). The time for acquiring the permutation test statistics is not considered. The calculations were performed on a pentium 4 CPU (2.80 GHz) with 0.98 GB of main memory.