Acta Geophysica vol. 59, no. 4, Aug. 2011, pp. 728-747 DOI: 10.2478/s11600-011-0013-5
Efficient Testing of Earthquake Forecasting Models
1
op
y
David A. RHOADES1, Danijel SCHORLEMMER2,3, Matthew C. GERSTENBERGER1, Annemarie CHRISTOPHERSEN1, J. Douglas ZECHAR4, and Masajiro IMOTO5
A ut ho
rc
GNS Science, New Zealand; e-mail:
[email protected] (corresponding author) 2 Southern California Earthquake Center, University of Southern California, Los Angeles, USA; e-mail:
[email protected] 3 GFZ German Research Centre for Geosciences, Potsdam, Germany 4 Swiss Seismological Survey, ETH Zurich, Zurich, Switzerland e-mail:
[email protected] 5 National Research Institute for Earth Science and Disaster Prevention (NIED), Tsukuba, Japan; e-mail:
[email protected] Abstract
Computationally efficient alternatives are proposed to the likelihood-based tests employed by the Collaboratory for the Study of Earthquake Predictability for assessing the performance of earthquake likelihood models in the earthquake forecast testing centers. For the conditional L-test, which tests the consistency of the earthquake catalogue with a model, an exact test using convolutions of distributions is available when the number of earthquakes in the test period is small, and the central limit theorem provides an approximate test when the number of earthquakes is large. Similar methods are available for the R-test, which compares the likelihoods of two competing models. However, the R-test, like the N-test and L-test, is fundamentally a test of consistency of data with a model. We propose an alternative test, based on the classical paired t-test, to more directly compare the likelihoods of two models. Although approximate and predicated on a normality assumption, this new T-test is not computer-intensive, is easier to interpret than the R-test, and becomes increasingly dependable as the number of earthquakes increases. Key words: statistical seismology, earthquake forecasting, earthquake likelihood models.
________________________________________________ © 2011 Institute of Geophysics, Polish Academy of Sciences
TESTING OF EARTHQUAKE FORECASTING MODELS
1.
729
INTRODUCTION
A ut ho
rc
op
y
The methods adopted by the earthquake forecast testing centers of the Collaboratory for the Study of Earthquake Predictability (CSEP; Jordan 2006, Zechar et al. 2010b) for real-time testing of earthquake forecasting methods require numerous synthetic earthquake catalogues to be generated from the expected number of earthquakes in each of many bins defined by time, magnitude and location to determine statistical significance in tests of each model. The primary tests adopted are the so-called N-test of the number of earthquakes predicted by the model, the L-test of the likelihood of the earthquake catalogue under the model, and the R-test of the relative likelihoods of two competing models (Schorlemmer et al. 2007). The efficiency of the calculations is an important issue, particularly for one-day forecasts which make the greatest demand on computer processing and storage. If the forecasts are updated and assessed daily, the processing burden may be manageable, even with inefficient methods. However, if the tests for many one-day periods have to be run retrospectively, then the use of inefficient methods is a significant drawback. For the R-test, the computational burden is proportional to the square of the number of models being tested. Therefore, to accommodate a large number of models in these tests, the procedures used should be as efficient as possible. Broadly speaking, there are two issues here. The first is that when synthetic catalogues are generated, the most efficient method available should be used to do it. The second is that such computer-intensive simulation methods should be used only when necessary, not when an equivalent exact test or a more efficient and sufficiently accurate approximate test could be used. Therefore, it is worth considering whether the intent of the tests presently implemented could be exactly or approximately accomplished more efficiently in some other way, and it is also important to quantify possible differences. The N-test, as originally described by Schorlemmer et al. (2007), involved extensive simulations as described above, but Zechar et al. (2010a) pointed out that the same test can be accomplished without such simulations, simply by comparing the observed number of earthquakes with quantiles of the distribution of the number of earthquakes forecast by the model (assuming Poisson uncertainty). This more efficient approach is now implemented in the CSEP testing software. Also, for the L-test, a more efficient way of generating synthetic catalogues has now been adopted than that proposed by Schorlemmer et al. (2007). This involves first simulating the number of earthquakes, and then using the relative bin expectations to assign each simulated earthquake to a particular bin (Zechar et al. 2010a).
730
D.A. RHOADES et al.
2.
rc
op
y
Imoto and Rhoades (2010) presented analytical methods for approximating the mean and variance of the log likelihood and log likelihood-ratio between two models to compute conventional normal approximations to the L-test and R-test. Imoto et al. (2011) extended this approach to allow for uncertainties in observed earthquake parameters. Their method has the advantage that it does not involve synthetic catalogues, but they assume that at most one earthquake occurs in each bin and therefore the binomial distribution can be used to approximate the means and variances. This condition is not necessarily satisfied in practice. For example, it was violated in the first half of the 5-year tests of regional earthquake likelihood models in California (Schorlemmer et al. 2010). In this paper we consider other efficient ways of computing the L- and R-tests. In applying the R-test to a retrospective analysis of five-year models submitted to the CSEP earthquake forecast testing centre in New Zealand, Gerstenberger et al. (2009) found that models sometimes mutually reject each other in the R-test. In such circumstances, the R-test becomes difficult to interpret. We therefore consider here a more straightforward alternative to the R-test which is both easier to interpret and involves far less computation. NOTATION AND THEORY
A ut ho
A regional earthquake likelihood model Λ is defined by the expected number of earthquakes {λ(i) , i = 1, ..., n} in each of a large number n of bins specified by limits in magnitude, location and time. These expected values are assumed to be the means of independent Poisson random variables (Schorlemmer et al. 2007). The log likelihood L of a catalogue Ω under the model Λ is then given by n
L ( Ω | Λ ) = ∑ [ω(i ) lnλ(i ) − λ(i ) − lnω(i )!] ,
(1)
i=1
where ω(i) is the number of earthquakes occurring in the ith bin. Schorlemmer et al. (2007) described statistical tests to be carried out with the aid of synthetic earthquake catalogues generated by simulating, in each bin, a Poisson random variable with the specified expected value. These tests are known as the N-test, L-test and R-test.
2.1 The L-test To establish whether or not the observed catalogue is consistent with the model, the L-test of Schorlemmer et al. (2007) compares the likelihood of the earthquake catalogue under the model with the distribution of likelihoods of synthetic earthquake catalogues conforming to the model.
TESTING OF EARTHQUAKE FORECASTING MODELS
731
Suppose that the N observed earthquakes occur in bins {ik, k = 1, …, N}, where the numbers ik are not necessarily all different. We denote the total number of earthquakes expected by Nˆ , i.e., n
Nˆ = ∑ λ (i ) ,
(2)
i =1
where n is the number of forecast bins. The log likelihood in eq. (1) can then be written as
y
N
L(Ω | Λ)= ∑ ⎣⎡ln λ (ik ) − ω (ik ) −1 ln ω (ik )!⎦⎤ − Nˆ .
(3)
op
k=1
rc
The second term in the summation is zero except where two or more earthquakes occur in the same bin. If λi
