METEOROLOGICAL APPLICATIONS Meteorol. Appl. 15: 85–95 (2008) Published online in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/met.55
Searching for systematic location errors of quantitative precipitation forecasts over the Calabria region Nazario Tartaglione,a * Stefano Mariani,b,c Marco Casaioli,c Christophe Accadia,d Stefano Federicoe,f and Silas Chr. Michaelidesg a Department of Physics, University of Camerino, Camerino, Italy Agency for Environmental Protection and Technical Services (APAT), Rome, Italy c Department of Mathematics, University of Ferrara, Ferrara, Italy d EUMETSAT, Darmstadt, Germany e CRATI consortium, Lamezia Terme, Italy Institute of Atmospheric Sciences and Climate – Italian National Research Council (ISAC – CNR), Section of Lecce, Lecce, Italy g Cyprus Meteorological Service, Nicosia, Cyprus b
f
ABSTRACT: This article statistically analyses the location errors of the precipitation patterns forecast by three limited area models, namely the Fifth-Generation NCAR/Penn State Mesoscale Model (MM5), the QUADRICS BOlogna Limited Area Model (QBOLAM) and the Regional Atmospheric Modelling System (RAMS), over the Calabria region (Italy) for the period October 2000–May 2002. Contiguous rain area (CRA) analysis is the diagnostic tool used to assess and quantify the position errors of the precipitation forecasts with respect to the observed precipitation patterns. Observation gridded analyses were obtained by means of the Barnes algorithm on the available rain gauge observations. Moreover, an approach to measure the quality of precipitation forecasts routinely by means of a global indicator called CRA Mean Shift (CMS) that summarizes the CRA verification outcomes is proposed. The CMS index would represent a statistical indicator of model quality in forecasting the correct positions of precipitation patterns. The model’s tendency to misplace the forecast precipitation patterns towards a particular direction was tested by using a bootstrap procedure. All models seem to show statistically poor abilities in forecasting the correct precipitation pattern position over the verification domain considered. As far as the tendency towards a particular direction is concerned, only the RAMS model seems to show a systematic horizontal misplacement of precipitation patterns towards a particular direction. Copyright 2008 Royal Meteorological Society KEY WORDS
verification; precipitation; forecast
Received 4 September 2007; Revised 14 December 2007; Accepted 2 January 2008
1.
Introduction
Population and industrial settlements are often concentrated in particular areas that can be particularly sensitive to high or even moderate precipitation. However, the prediction of the correct rainfall amount in specific locations is still a challenge, especially for warm season precipitation events (Gallus, 2002). Investigating systematic forecasting errors represents an important activity for improving the ability of numerical weather prediction (NWP) models to predict timing and location of rainfall well. Such systematic errors can be studied by means of verification techniques on long time series (of at least several months) of precipitation patterns. Verification of quantitative precipitation forecast/s (QPF/s) is a central task in meteorological centres (Mesinger, 1996; Wilson, 2001). In the past, researchers * Correspondence to: Nazario Tartaglione, Department of Physics, University of Camerino, Via Madonna delle carceri, 9, I-62032 Camerino, Italy. E-mail:
[email protected] Copyright 2008 Royal Meteorological Society
focussed mainly on verification methods based on comparison of forecasts and observations at multiple points, these methods are well described in the literature (e.g. Wilks, 1995; Jolliffe and Stephenson, 2003, and references therein). One of the drawbacks of such an approach is the impossibility of directly extracting information on forecast location errors. Furthermore, continuous measures such as mean square error (MSE) and correlation coefficient or, more generally, contingency table-based measures, may give poor skill scores when a spatial rainfall forecast is correct in intensity, but incorrect in position, possibly resulting in double penalty effects. Only recently have more sophisticated approaches been introduced to account for the spatial displacement of rainfall patterns (e.g. Hoffman et al., 1995; Ebert and McBride, 2000, hereafter referred to as EMB). The EMB method is referred to as contiguous rain area (CRA) analysis. The displacement error is measured by translating the rainfall patterns forecast over the observed
86
N. TARTAGLIONE ET AL.
pattern until a best-fit criterion, such as the minimization of MSE, is satisfied. Although the application of the CRA analysis has been used mainly over Australia (EMB) and USA (Grams et al., 2006), some of the authors of the present work have applied this technique in some verification studies applying the CRA analysis over the Mediterranean area (e.g. Mariani et al., 2005; Tartaglione et al., 2005). The purpose of the present study is to estimate the systematic location errors, if any, of three numerical limited area model/s (LAM/s) running over the Calabria region (Figure 1), located at the southernmost tip of the Italian peninsula. This region is characterized by steep orography. As shown in the next sections, the direct outcome of this verification exercise is not easy to interpret because of the large amount of values. Consequently, a statistical index that summarizes the CRA outcomes is
introduced. The values of this index are then compared with the values of the same index obtained by assuming a Gaussian distribution of forecast position errors to define an expected value for this indicator. Furthermore, it is investigated whether the forecasts show systematic displacement errors in a preferential direction, and this hypothesis is tested using a bootstrap procedure. This work was started in late 2002 in the context of an operational-oriented collaboration among the Agency for Environmental Protection and Technical Services (APAT), the Institute of Atmospheric Sciences and Climate–Italian National Research Council (ISAC–CNR) and the Consorzio per la Ricerca e le Applicazioni di Tecnologie Innovative (CRATI). The first aim was a QPF verification of two LAMs, namely the Regional Atmospheric Modelling System (RAMS) and the FifthGeneration NCAR/Penn State Mesoscale Model (MM5),
Figure 1. Geo-localization of Calabria: (a) Mediterranean Basin (screenshot from NASA’s globe software World Wind) with indication of Italy and Calabria region (within the box); (b) Contours of Calabria coastline (thick line) and orography. Contours with a 500 m interval. The zero contour is omitted. Copyright 2008 Royal Meteorological Society
Meteorol. Appl. 15: 85–95 (2008) DOI: 10.1002/met
PRECIPITATION DISPLACEMENT ERRORS OVER THE CALABRIA REGION
87
set-up by the CRATI consortium to produce weather forecasts for Calabria to be provided to the regional civil protection service (Federico et al., 2004). In that previous work, QPF verification was performed by means of standard methods: the contingency table-based nonparametric skill scores (Wilks, 1995), such as the frequency bias score (BIA), the equitable treat score (ETS) and the Hanssen-Kuipers score (HK), where uncertainties in skill score differences were assigned using the bootstrap technique proposed by Hamill (1999). In addition, precipitation fields forecast by the APAT’s QUADRICS Bologna Limited Area Model (QBOLAM) were included in the inter-comparison as they were available for the same period. Subsequently, it was decided to extend the initial verification study by implementing the CRA analysis at a regional level. The dataset and the region were used as a test-bed for a possible operational application of the CRA analysis over complex terrain, which has sufficiently dense rain gauge coverage. The dataset is described in Section 2 together with a short description of the models used. The procedure employed for the verification task is described in Section 3. Section 4 discusses the location errors of the three models. Conclusions are drawn in Section 5. 2.
Data and models
2.1. Observed data and interpolation process Rainfall data were obtained by rain gauge networks belonging to APAT (formerly the Servizio Idrografico e Mareografico Nazionale–SIMN network; 73 gauges), the Regional Service of Sicily (7 gauges) and the CRATI consortium (4 gauges), as shown in Figure 2(a). The common grid used for the verification exercise has a grid box size of 0.1° in both latitude and longitude (Figure 2(b)); hence, in order to produce an adequate (i.e. less sensitive to grid box size) observed rainfall analysis, a two-pass Barnes objective analysis scheme (Barnes, 1964, 1973; Koch et al., 1983) was employed. The Barnes scheme assigns to each gauge observation a Gaussian weight as a function of distance between the gauge and the grid box centre. A first pass is performed to produce a first-guess precipitation analysis, whilst the second pass increases the amount of detail of the prior pass. Specifically, rainfall data were collected for 608 days, from 1st October 2000 to 31st May 2002 (see Figure 3 for the daily number of working stations). The Barnes procedure was applied setting the average data spacing to 0.2° , a setting consistent with the constraint that the ratio between grid size and average data spacing lies between 0.3 and 0.5 (the shortest wave that can be resolved by the rain gauge network). The convergence parameter was set to 0.2, which produced the maximum gain in detail with the proposed design. In order to avoid excessive rainfall spreading introduced by the analysis scheme on grid points far from the actual locations of gauges, those grid points that did not have any rain Copyright 2008 Royal Meteorological Society
Figure 2. Verification area: (a) Geographical distribution of the rain gauges, mainly located in Calabria and in some parts of Basilicata, Puglia and Sicily, which were employed in the verification. (b) 0.1° verification grid, which spans an area about 300 km east–west by 300 km north–south.
Figure 3. Time series of the number of the working rain gauge stations during the 20-months observational period (608 days).
gauge within a radius of 0.15° were considered in the precipitation gridded analysis as data-void points. 24 h accumulation was chosen as the temporal scale for the rainfall verification, so gridded observation fields were accumulated on a daily basis, starting from 0000 UTC. As a consequence, observations were less prone to representativeness errors, as also indicated by Meteorol. Appl. 15: 85–95 (2008) DOI: 10.1002/met
88
N. TARTAGLIONE ET AL.
WWRP/WGNE (2004). Moreover, it was easier to identify, over the small verification domain, the observed (and the forecast) precipitation patterns to be used in the CRA analysis. 2.2.
Numerical models
Three models were used in the present study: QBOLAM that runs operationally at APAT, MM5 and RAMS that run operationally at the CRATI consortium. The model domains are shown in Figure 4.
QBOLAM is a hydrostatic model, which is derived from the well-known BOlogna Limited Area Model (BOLAM; Buzzi et al., 1994; Buzzi and Foschini, 2000), and it is part of the Sistema Idro-Meteo-Mare (SIMM), a hydro-meteo-marine forecasting system developed for the operative needs of APAT (Speranza et al., 2004, 2007). Recently, the forecasting system has been ported onto a Linux cluster Altix 350. However, for this study, the previous QBOLAM version, which runs on a Quadrics supercomputer, was used. Here, the main characteristics of this version are singled out. The horizontal grid
Figure 4. Model domains. (a) The three domains of the Fifth-Generation NCAR/Penn State Mesoscale Model (MM5): GRID #1 (solid line) at 54 km resolution; GRID #2 (dot-dashed line) at 18 km resolution; and GRID #3 (dashed line) at 6 km resolution. (b) The two domains of the QUADRICS BOlogna Limited Area Model (QBOLAM): GRID #1 (solid line) at 30 km resolution; and GRID #2 (dashed line) at 10 km resolution. (c) The two domains of the Regional Atmospheric Modelling System (RAMS): GRID #1 (solid line) at 30 km; and GRID #2 (dashed line) at 6 km. Copyright 2008 Royal Meteorological Society
Meteorol. Appl. 15: 85–95 (2008) DOI: 10.1002/met
PRECIPITATION DISPLACEMENT ERRORS OVER THE CALABRIA REGION
89
spacing is 0.1° (see Figure 4(a)) and the vertical coordinate is expressed in sigma coordinates (40 levels). The advection scheme is the forward-backward advection scheme (FBAS) developed by Malguzzi and Tartaglione (1999). The convection scheme is Kuo’s (1974), and the radiation scheme is Page’s (1986). These schemes were chosen for their simplicity in implementation of physical parameterizations on the QUADRICS computer (Speranza et al., 2004, 2007). Initial and boundary conditions were provided by a coarser (0.3° horizontal grid spacing) version of the same model, which in turn used initial and boundary conditions from ECMWF. MM5 is a non-hydrostatic model which is fully described in Grell et al. (1994). The MM5 simulations used in the study have been produced over the 6 km grid domain (inner grid, see Figure 4(c)) with 24 full sigma levels, unevenly spaced, in the vertical with maximum resolution in the PBL. MM5 employs the explicit moisture scheme by Hsie et al. (1984), with improvements, as reported in Dudhia et al. (1999), in order to allow for ice-phase microphysics below 0 ° C. The Kain-Fritsch cumulus parameterization (Kain and Fritsch, 1990) is applied to the two outer domains (see Figure 4(c)) with grid resolutions of about 54 and 18 km, respectively. The RAMS model is described by Pielke et al. (1992) and Cotton et al. (2003). The version of RAMS used for this study has a 6 km horizontal grid spacing (inner grid, see Figure 4(b)) and 25 vertical levels, up to 12 500 m in a terrain-following coordinate system. Parameterization of fluxes between atmosphere and surface is described in Walko et al. (2000). Seven types of hydrometeors, together with water vapour, are used in the parameterization of large-scale (stratiform) precipitation. Convective precipitation is parameterized using the Molinari and Corsetti (1985) approach, which is a simplified form of the Kuo scheme that accounts for updrafts and downdrafts. Initial and dynamic boundary conditions, for both MM5 and RAMS models, are from 1200 UTC ECMWF analyses and forecasts. For the comparison, precipitation forecasts were accumulated over 24 h from T + 12 to T + 36. In previous work by Federico et al. (2004), in which details of how precipitation is operatively forecast by MM5 and RAMS are explained, models were verified considering forecast outputs over their native grids, whilst in the present study, forecast data were post-processed on a common 0.1° verification grid (Figure 2(b)) by means of a simple nearest-neighbour average method, also known as re-mapping. A thorough review of the remapping technique is provided by Accadia et al. (2003).
method is based on a pattern match of two CRAs, the observed and forecast entities, delimited by a chosen isohyet (also referred to as the CRA rain contour). The assessment of the best agreement between the observed and forecast entities is obtained by shifting the forecast pattern, in longitude and latitude, within a rectangular domain enclosing the two CRAs. The maximum allowed shifting value, which is arbitrarily selected, has been set to 5 grid points, that is, the forecast entity is shifted in the two directions from −5 × 0.1° to 5 × 0.1° . Over a small domain, too large a maximum shifting value might produce unreliable results by satisfying the best-fit criterion with patterns that are not physically associated with observation (Tartaglione et al., 2005). Three pattern-matching criteria were used in the CRA analysis, namely, the correlation maximization, the mean absolute error (MAE) minimization and the MSE minimization. Furthermore, in order to achieve a reasonable pattern match between the observed and forecast entities, a minimum correlation threshold must be exceeded. The value depends on the effective number of samples compared, which is a function of the number of grid points where the analysis is performed (which changes from shift to shift), and the autocorrelation of both the observed and forecast fields (Xie and Arkin, 1995). Thus, the F-test (Panofsky and Brier, 1958), with a 95% confidence level, was chosen to assess the statistical significance of each shift. This means that in the CRA analysis those shifts which are not statistically significant were not considered.
3.
Before discussing the location errors of models, some limitations of the present study should be noted. One is related to the spatial distribution of rain gauges. The presence or absence of rain gauges in some areas might change the location error statistics. Another limitation is associated with the form and relative size of the
Comparison procedure
3.1. The contiguous rain area analysis In order to compare results obtained by the three LAMs and quantify possible systematic location errors in the forecast fields, the CRA analysis was applied. The Copyright 2008 Royal Meteorological Society
3.2. Events selection The CRA analysis was performed only on a data subset, with a sample size of about 200 days out of 608 days (Figure 3), from October 2000 to May 2002. Sample size changes slightly from model to model. The sample size is a function of the CRA rain contour selected for the analysis (for this work an isoyet of 0.5 mm day−1 was chosen) and it was obtained excluding: 1. no-rain days (both observed and forecast); 2. days with an insufficient number of comparison points, that is, a number insufficient to identify both the observed and the forecast entities correctly (i.e. less than 5 grid points); 3. days for which a statistically significant shift (including the no-shift case) was not found; this means that the correlation between observed and forecast fields was too low, with respect to the number of comparison grid points, in order to assess statistically the pattern match among the two selected entities in a significant way.
Meteorol. Appl. 15: 85–95 (2008) DOI: 10.1002/met
90
N. TARTAGLIONE ET AL.
region considered. The Calabria region is a very narrow peninsula. These limitations might actually affect the quality of the output from the CRA analysis technique. The methodology developed in this paper is considered to be generally applicable for the assessment of the quality of precipitation forecasts from numerical models.
4. 4.1.
Location errors Observed errors
This section describes the results of the CRA analysis. Figure 5 shows the distribution of a number of events
as a function of longitudinal and latitudinal displacements needed to minimize MSE (Figure 5(a), (b), (c)), MAE (Figure 5(d), (e), (f)) and to maximize correlation (Figure 5(g), (h), (i)). In other words, each number in the 11 × 11 matrices of Figure 5 represents the number of events for which the precipitation pattern needed to be shifted a certain longitudinal (x axis) and latitudinal (y axis) displacement to satisfy the chosen best-fit criterion. The central point, corresponding to (0° , 0° ) coordinates, represents the noerror situation. For this case, events that did not need to be shifted to satisfy the best-fit criterion were summed and the total was placed in the central position of matrices, so obtaining the number of events with no location errors
Figure 5. CRA precipitation results using as pattern-matching criterion either the minimization of MSE (a, b, c), the minimization of MAE (d, e, f), or the maximization of correlation (g, h, i) for Fifth-Generation NCAR/Penn State Mesoscale Model – MM5 (a, d, g), QUADRICS BOlogna Limited Area Model – QBOLAM (b, e, h) and Regional Atmospheric Modelling System – RAMS (c, f, i). The density of shading indicates the number of best matches within each cell. Copyright 2008 Royal Meteorological Society
Meteorol. Appl. 15: 85–95 (2008) DOI: 10.1002/met
PRECIPITATION DISPLACEMENT ERRORS OVER THE CALABRIA REGION
(see Table I). The other numbers within the matrices were obtained in the same way but referred to a particular displacement among the 121 (11 × 11) possible ones. The outcomes from the verification process performed for the three models, MM5 (Figure 5(a), (d), (g)), QBOLAM (Figure 5(b), (e), (h)) and RAMS (Figure 5(c), (f), (i)), show only a small proportion of events with location errors smaller than 0.1° (Table I). As the number of possible displacements (121) is similar to the total number of considered events (about 200) a very high number for each grid box is not expected. A good model for predicting precipitation patterns is expected to have a high number of no-error events, or at least errors concentrated near the (0, 0) position. The MM5 model, despite having the maximum number of events with no errors, also shows the highest number of events (10 w.r.t. MSE–Figure 5(a), and
91
11 w.r.t. MAE–Figure 5(d)) in a single grid box placed far enough [coordinates (0.2° , −0.5° )] from the central position. RAMS and QBOLAM show fewer events than MM5 (with no error) with respect to the MSE and MAE criteria. However, they improve when the correlation criterion is used, and they seem to be better than MM5. Simply analysing the central point alone does not adequately characterize the quality of the numerical model. Thus, in order to assess the quality of the numerical model, it is necessary to analyse the entire distribution of the displacement errors. Extracting immediate information from a visual inspection is difficult because of the large amount of data involved. Moreover, the results should be compared with a model describing the error expectation. Such a model will be discussed in the next subsection.
Figure 5. (Continued). Copyright 2008 Royal Meteorological Society
Meteorol. Appl. 15: 85–95 (2008) DOI: 10.1002/met
92
N. TARTAGLIONE ET AL.
The frequency depends on the model and criterion used. N is the maximum displacement (in grid boxes, in this study N is equal to 5) used during the application of the CRA analysis. The term wij measures the distance from the centre of the matrix (0, 0), which indicates no location error; it is simply formulated in the following way: wij =
Figure 5. (Continued).
Table I. The number (absolute and relative values) of daily events with no location errors is reported for each of the three contiguous rain area (CRA) best-fit criteria. Relative values, which are expressed as a percentage with respect to total number of the events, were rounded-off to 1 decimal place. Model
CRA criterion MSE
MM5 QBOLAM RAMS
4.2.
MAE
CORR
Abs
Rel %
Abs
Rel %
Abs
Rel %
8 2 5
3.7 1.1 2.6
7 1 4
3.2 0.6 2.0
3 4 6
1.5 2.4 3.3
A paradigm for the location error
As said above, the number of events with no location errors can be important, but it cannot be a reliable indicator of the quality of a model in terms of the displacements. It is possible, for example, that a model is able to forecast the precipitation pattern well for many events, but when it is wrong the location error can be large. Thus, an indicator describing quantitatively the location error is needed; it should also take into account all displacements on long time series. At this point, we could measure goodness-of-position pattern forecast in terms of distance from the centre of the matrices shown in Figure 5. Accordingly, the CRA Mean Shift (CMS), defined as: CMS =
N N
fij · wij
(1)
i=−N j =−N
is introduced, so that CMS is proportional to the frequency fij of shifts with respect to the number of events for each grid box represented by the subscripts i and j . Copyright 2008 Royal Meteorological Society
i2 + j 2
(2)
√ The CMS ranges between 0 and N 2; where a CMS equal to zero indicates a perfect forecast in√ terms of spatial error, whereas a CMS equal to N 2 means that the forecast is misplaced far away with respect to the gridded analysis. Although such a number can be normalized dividing by N , it is preferred to maintain the non-normalized form since by multiplying for 0.1° the grid step of the verification domain we can determine the MAE in degrees and consequently in kilometres. At latitudes of the Calabria region, 0.1° corresponds to about 10 km horizontally (11 km in latitude and 8.5 km in longitude). The values of CMS, obtained by using the chosen models and criteria, appear in Table II. As can be seen in the Table, the CMS is about 4 for all situations considered, indicating a mean error of little more than 40 km in longitude and 50 km in latitude. The analysis is extended to assess whether this error is characterized by a preferred direction (Section 4.3). If the models have no systematic position error and uncorrelated events are being considered, the position errors will be independent. Therefore, it is expedient to regard position forecasts as good if the distributions of location errors are Gaussian-like with a small standard deviation. For a perfect model, the CMS would be zero and the Gaussian would be really a Dirac-delta. If the location errors are distributed as a bivariate Gaussian function, the CMS takes different values depending on the standard deviation of the Gaussian. The variation of CMS as a function of the standard deviation for a Gaussian distribution is shown in Figure 6. Hence, Figure 6 shows the behaviour of CMS when errors are normally distributed over the 11 × 11 verification domain with the standard deviation value going from 1 up to 10. In practical terms, distributions having standard deviations approaching a value of 10 will have a CMS approaching 4. This does not come as a surprise because when the standard deviation becomes larger the Gaussian probability distribution function, on a limited domain, tends to a uniform distribution, which has a CMS equal to 4.2. A value of the CMS (Figure 6) less than 2.5 (before the plateau) would be acceptable as describing a good ability of the model to predict precipitation patterns correctly. This is because the plateau indicates that the Gaussian distribution is becoming a uniform distribution, and the number of events is scattered over the error matrix instead of remaining concentrated close to the (0, 0) point. Unfortunately, all situations examined here and presented Meteorol. Appl. 15: 85–95 (2008) DOI: 10.1002/met
93
PRECIPITATION DISPLACEMENT ERRORS OVER THE CALABRIA REGION
Table II. Summary of the verification outcomes reported for each model with respect to the three contiguous rain area (CRA) best-fit criteria. Over a period of 608 days, the CRA analysis is performed over a data subset (between 165 and 202 days, see the ‘number of events’ column). No-rain days and days either with an insufficient number of comparing points or for which a statistically significant CRA shift was not found are excluded. The CRA Mean Shift (CMS) value gives a global evaluation of verification. The northeast, southeast, southwest, northwest shifts represent the frequencies of events that lay in the sub-domains in which the verification domain was divided (see Figure 7). The directions having the higher frequencies (preferential directions for position errors) are indicated in boldface. Model
MM5 MM5 MM5 QBOLAM QBOLAM QBOLAM RAMS RAMS RAMS
Matching criterion
Number of events
CRA Mean shift (CMS)
Northeast shift (%)
Southeast shift (%)
Southwest shift (%)
Northwest shift (%)
CORR MAE MSE CORR MAE MSE CORR MAE MSE
202 217 217 165 174 174 182 196 196
4.030 3.987 4.101 3.995 4.437 4.486 4.050 4.282 4.340
19.80 24.42 23.50 27.27 22.99 25.29 23.08 20.92 22.45
28.22 26.27 26.73 27.21 25.99 22.99 29.12 32.14 29.59
19.80 19.35 20.74 21.82 25.29 25.29 19.78 19.39 21.43
30.69 26.27 25.81 22.42 25.29 25.86 24.73 25.00 24.49
Figure 6. Values of the Contiguous Rain Area Mean Shift (CMS) for the location errors distributed as bivariate Gaussian over the considered shift domain, as a function of the standard deviation.
in Figure 5 and summarized in Table II show too high a value of the CMS index. A χ 2 test confirms that the null hypothesis of a Gaussian distribution with a standard deviation less than or equal to 5 is always rejected, with 95% significance, for all cases examined in the present paper. The success of the χ 2 test was obtained for 2 out of 9 analysed cases with a high value (around 10) of the standard deviation, confirming that data in the matrices are, at least for those cases, more likely to be uniformly distributed. From the point of view of the CMS, the numbers of events with position errors are scattered over the verification domain, and some models seem to show a preferential direction. It is now appropriate to investigate whether such a kind of error along a particular direction is systematic. 4.3. Direction of location error In order to investigate whether a preferential direction exists for the found position errors, the verification domain is divided into four areas, as shown by Figure 7. The choice of dividing the domain in that manner was Copyright 2008 Royal Meteorological Society
Figure 7. Sub-domains determining preferential directions of forecast precipitation patterns. The sub-domains on top-left and top-right are respectively northwest (dashed line) and northeast (solid line). The areas on bottom-left and bottom-right are respectively southwest (long-dashed line) and southeast (dotted line).
made so as to have an equal number of grid boxes for each sub-domain. Moreover, by dividing it into four parts, each sub-domain has a relatively high number of the grid boxes. The notations used to indicate the sub-domains use compass directions. For each examined case (model and best-fit criterion), the percentage of the events belonging to each subdomain is shown in Table II. QBOLAM’s errors do not show a preferred direction. RAMS errors seem to be preferably distributed along the southeast direction, whereas MM5 errors show the maximum percentage along the southeast and northwest directions. It is interesting to know whether percentage differences between each pair of directions are actually so Meteorol. Appl. 15: 85–95 (2008) DOI: 10.1002/met
94
N. TARTAGLIONE ET AL.
high because models have a real tendency to shift their precipitation forecasts in a preferred direction, or if they are obtained by chance. In order to investigate whether the differences between directions were significant or not, we sampled the longitudinal and latitudinal shifts. These were independently sampled with a bootstrap technique 10 000 times for each examined case. Investigation is limited to differences between observed percentages rather than the absolute values, and in particular, to differences between the maximum observed percentage (written in boldface in Table II) with other percentages. MM5-CORR has a percentage of events that are shifted towards the northwest that is significantly different, at the 95% confidence, with respect to the northeast and southwest. In contrast, RAMS-MAE has a percentage of events towards southeast that is significantly different at the 95% confidence level, with respect to northeast and southwest directions. As already noted, QBOLAM shows little evidence of a preferred direction and this is confirmed by the bootstrap technique. The MM5 model shows a preferred direction for location errors only when the correlation criterion is used. Differently from QBOLAM and MM5, RAMS seems to produce precipitation patterns having a preferential direction with respect to the observations, along the southeast–northwest axis, with displacements located above all on the southeast part. This claim is suggested by observing that differences between the southeast direction and northeast and southwest, are statistically significant with confidence intervals going up from 90% to more than 95% for all considered cases, as shown in Table III.
5.
been used: the minimization of MSE; the minimization of MAE; and the maximization of correlation. The numbers in the matrices of Figure 5, representing the CRA outcomes, are representative of how many events the forecast pattern has to be displaced by a specific amount with respect to the observed pattern. At first glance, a clearly defined value of the error cannot be evaluated. Moreover, the distribution of events with a specific error on the error space changes as a function of criterion used. This is particularly true when results obtained by MAE and MSE criteria are compared with those obtained using the correlation criterion. It might be thought that the proportion of events with no location error might provide a useful indication of the model quality, however, in practice only a small fraction of forecasts fall into this category, so it is necessary to develop an objective measure which takes into account the total distribution of location errors. Thus, an index (CMS) has been introduced to estimate the mean location error using Equation (1). For all models and criteria used (see Table II) the CMS is about 4, indicating a mean position error of a little more than 40 km in a 24 h forecast. It can be reckoned that the models show poor ability to forecast position patterns correctly, relying on a Gaussian assumption of the theoretical displacement distribution that, moreover, our outcomes seem not to show. The hypothesis that some models have a systematic displacement error bias in certain directions was tested. It was found that only the RAMS model has a systematic error bias (in the southeast direction), whereas MM5 and QBOLAM did not. However, the MM5 model showed slight evidence of an error bias in the northwest direction, but only when correlation was used as a best-fit criterion.
Conclusions
The systematic location errors of precipitation forecasts obtained by three models covering Calabria, an Italian region, have been estimated. The position error statistics have been derived by the CRA analysis applied to 200 precipitation events, which occurred in Calabria. The CRA determines the location error obtained by translating the forecast pattern over the observed one in order to satisfy a best-fit criterion. Three separate criteria have Table III. Statistical significance obtained by using a bootstrap technique, of the differences between the preferential directions (in boldface in Table II and with a hyphen here) and the other ones. Only outcomes with significance more than or equal to 90% are reported. A ‘NO’ flag is used to indicate that the difference is not significant. Model
MM5 RAMS RAMS RAMS
Matching criterion
Northeast
Southeast
Southwest
Northwest
CORR CORR MAE MSE
95% 90% 95% 90%
NO – – –
95% 90% 95% 90%
– NO NO NO
Copyright 2008 Royal Meteorological Society
Acknowledgements The authors thank APAT’s Department of Inland and Marine Waters Protection (formerly SIMN) for the rain gauge and the QBOLAM forecast data. They also thank the CRATI consortium, and in particular, Mr Elenio Avolio, for rain gauges, and MM5 and RAMS forecast data. Moreover, Regione Sicilia is acknowledged for providing rainfall data gathered by the Sicilian rain gauge network. We wish to thank Dr Elisabeth Ebert for her helpful comments during the ‘Third International Workshop on Verification Methods’ held at ECMWF in January 2007. Two anonymous reviewers are acknowledged for their valuable suggestions. Coastline data provided by National Oceanic and Atmospheric Administration; orography data provided by United States Geological Survey. References Accadia C, Mariani S, Casaioli M, Lavagnini A, Speranza A. 2003. Sensitivity of precipitation forecast skill scores to bilinear interpolation and a simple nearest-neighbor average method on highresolution verification grids. Weather and Forecasting 18: 918–932. Barnes SL. 1964. A technique for maximizing details in numerical weather map analysis. Journal of Applied Meteorology 3: 396–409. Meteorol. Appl. 15: 85–95 (2008) DOI: 10.1002/met
PRECIPITATION DISPLACEMENT ERRORS OVER THE CALABRIA REGION Barnes SL. 1973. Mesoscale objective analysis using weighted timeseries observations. NOAA Tech. Memo. ERL NSSL-62. National Severe Storms Laboratory: Norman, OK, 73069, 60, [NTIS COM73-10781.]. Buzzi A, Foschini L. 2000. Mesoscale meteorological features associated with heavy precipitation in the southern Alpine region. Meteorology and Atmospheric Physics 72: 131–146. Buzzi A, Fantini M, Malguzzi P, Nerozzi F. 1994. Validation of a limited area model in cases of Mediterranean cyclogenesis: Surface fields and precipitation scores. Meteorology and Atmospheric Physics 53: 53–67. Cotton WR, Pielke RA Sr, Walko RL, Liston GE, Tremback CJ, Jiang H, McAnelly RL, Harrington JY, Nicholls ME, Carrio GG, McFadden JP. 2003. RAMS 2001 current status and future directions. Meteorology and Atmospheric Physics 82: 5–29. Dudhia J, Gill D, Guo Y-R, Hansen D, Manning K, Wang W. 1999. PSU/NCAR Mesoscale Modeling System Tutorial Class Notes and Users’ Guide (MM5 Modeling System Version 2 with an introduction to version 3). [Available online at http://www.mmm.ucar.edu/ mm5/mm5-home.html]. Ebert EE, McBride JL. 2000. Verification of precipitation in weather systems: determination of systematic errors. Journal of Hydrology 239: 179–202. Federico S, Avolio E, Bellecci C, Colacino M, Lavagnini A, Accadia C, Mariani S, Casaioli M. 2004. Three models intercomparison for quantitative precipitation forecast over Calabria. Nuovo Cimento Della Societa Italiana Di Fisica C-Geophysics and Space Physics 27: 627–647. Gallus WA. 2002. Impact of verification grid-box size on warm-season QPF skill measures. Weather and Forecasting 17: 1296–1302. Grams JS, Gallus WA, Koch SE, Wharton LS, Loughe A, Ebert EE. 2006. The use of a modified Ebert-McBride technique to evaluate mesoscale model QPF as a function of convective system morphology during IHOP 2002. Weather and Forecasting 21: 288–306. Grell GA, Duhia J, Stauffer DR. 1994. A description of the fifthgeneration Penn State/NCAR mesoscale model (MM5). NCAR Technical note – NCAR/TN-398+STR NCAR: Boulder, CA. Hamill TM. 1999. Hypothesis tests for evaluating numerical precipitation forecasts. Weather and Forecasting 14: 155–167. Hoffman RN, Liu Z, Louis J-F, Grassotti C. 1995. Distortion representation of forecast errors. Monthly Weather Review 123: 2758–2770. Hsie E-Y, Anthes RA, Keyser D. 1984. Numerical simulation of frontogenesis in a moist atmosphere. Journal of the Atmospheric Sciences 41: 2581–2594. Jolliffe IT, Stephenson DB (Eds). 2003. Forecast Verification: A Practitioner’s Guide in Atmospheric Science. John Wiley and Sons: Chichester; 240. Kain JS, Fritsch JM. 1990. A one-dimensional entraining/detraining plume model and its application in convective parameterization. Journal of the Atmospheric Sciences 47: 2784–2802. Koch SE, Desjardins M, Kocin PJ. 1983. An interactive Barnes objective map analysis scheme for use with satellite and conventional data. Journal of Applied Meteorology 22: 1487–1503. Kuo HL. 1974. Further studies of the parameterization of the influence of cumulus convection on large scale flow. Journal of the Atmospheric Sciences 31: 1232–1240. Malguzzi P, Tartaglione N. 1999. An economical second order advection scheme for numerical weather prediction. Quarterly Journal of the Royal Meteorological Society 125: 2291–2303.
Copyright 2008 Royal Meteorological Society
95
Mariani S, Casaioli M, Accadia C, Llasat MC, Pasi F, Davolio S, Elementi M, Ficca G, Romero R. 2005. A limited area model intercomparison on the “Montserrat-2000” flash-flood event using statistical and deterministic methods. Natural Hazards and Earth System Sciences 5: 565–581. Mesinger F. 1996. Improvements in quantitative precipitation forecasting with the Eta regional model at the National Centers for Environmental Prediction The 48-km upgrade. Bulletin of the American Meteorological Society 77: 2637–2649. Molinari J, Corsetti T. 1985. Incorporation of cloud-scale and mesoscale down-drafts into a cumulus parametrization results of one and three-dimensional integrations. Monthly Weather Review 113: 485–501. Page JK. 1986. Prediction of Solar Radiation on Inclined Surfaces, Solar Energy R&D in the European Community Series F , Vol. 3. Dordrecht Reidel Publishing Company: Dordrecht, 459. Panofsky HA, Brier GW. 1958. Some Applications of Statistics to Meteorology. Pennsylvania State University: University Park, Pennsylvania; 224. Pielke RA, Cotton WR, Walko RL, Tremback CJ, Lyons WA, Grasso LD, Nicholls ME, Murran MD, Wesley DA, Lee TH, Copeland JH. 1992. A comprehensive meteorological modeling system-RAMS. Meteorology and Atmospheric Physics 49: 69–91. Speranza A, Accadia C, Mariani S, Casaioli M, Tartaglione N, Monacelli G, Ruti PM, Lavagnini A. 2007. SIMM: an integrated forecasting system for the Mediterranean area. Meteorological Applications 14: 337–350. Speranza A, Accadia C, Casaioli M, Mariani S, Monacelli G, Inghilesi R, Tartaglione N, Ruti PM, Carillo A, Bargagli A, Pisacane G, Valentinotti F, Lavagnini A. 2004. POSEIDON: an integrated system for analysis and forecast of hydrological, meteorological and surface marine fields in the Mediterranean area. Nuovo Cimento Della Societa Italiana Di Fisica C-Geophysics and Space Physics 27: 329–345. Tartaglione N, Mariani S, Accadia C, Speranza A, Casaioli M. 2005. Comparison of rain gauge observations with modeled precipitation over Cyprus using Contiguous Rain Area Analysis. Atmospheric Chemistry and Physics 5: 2147–2154. Walko RL, Band LE, Baron J, Kittel TG, Lammers R, Lee TJ, Ojima D, Pielke RA Sr, Taylor C, Tague C, Tremback CJ, Vidale PL. 2000. Coupled atmosphere-biosphere-hydrology models for environmental prediction. Journal of Applied Meteorology 39: 931–944. Wilks DS. 1995. In Statistical Methods in the Atmospheric, Dmowska R, Holton JR (eds). Science Academic Press: San Diego, CA; 467. Wilson C. 2001. Review of current methods and tools for verification of numerical forecasts of precipitation. COST717 Working Group Report on Approaches to verification. [Available online at http//wwwsmhise/cost717/doc/WDF 02 200109 1pdf]. WWRP/WGNE Joint Working Group on Verification. 2004. Recommendations for the verification and intercomparison of QPFs from operational NWP models. [Available online at http://www.bom.gov.au/bmrc/wefor/staff/eee/verif/WGNE/ QPF verif recomm.pdf]. Xie P, Arkin PA. 1995. An intercomparison of gauge observations and satellite estimates of monthly precipitation. Journal of Applied Meteorology 34: 1143–1160.
Meteorol. Appl. 15: 85–95 (2008) DOI: 10.1002/met
