A suggestion for computing objective function in ... - Semantic Scholar

Ecological Informatics 24 (2014) 107–111

Contents lists available at ScienceDirect

Ecological Informatics journal homepage: www.elsevier.com/locate/ecolinf

A suggestion for computing objective function in model calibration Yiping Wu a,⁎, Shuguang Liu b,⁎ a b

ASRC Federal, contractor to the U.S. Geological Survey (USGS) Earth Resources Observation and Science (EROS) Center, Sioux Falls, SD 57198, United States U.S. Geological Survey (USGS) Earth Resources Observation and Science (EROS) Center, Sioux Falls, SD 57198, United States

a r t i c l e

i n f o

Article history: Received 1 November 2013 Received in revised form 4 August 2014 Accepted 6 August 2014 Available online 13 August 2014 Keywords: Biogeochemical modeling Hydrological modeling Model calibration Objective function Parameter optimization

a b s t r a c t A parameter-optimization process (model calibration) is usually required for numerical model applications, which involves the use of an objective function to determine the model cost (model-data errors). The sum of square errors (SSR) has been widely adopted as the objective function in various optimization procedures. However, ‘square error’ calculation was found to be more sensitive to extreme or high values. Thus, we proposed that the sum of absolute errors (SAR) may be a better option than SSR for model calibration. To test this hypothesis, we used two case studies—a hydrological model calibration and a biogeochemical model calibration—to investigate the behavior of a group of potential objective functions: SSR, SAR, sum of squared relative deviation (SSRD), and sum of absolute relative deviation (SARD). Mathematical evaluation of model performance demonstrates that ‘absolute error’ (SAR and SARD) are superior to ‘square error’ (SSR and SSRD) in calculating objective function for model calibration, and SAR behaved the best (with the least error and highest efficiency). This study suggests that SSR might be overly used in real applications, and SAR may be a reasonable choice in common optimization implementations without emphasizing either high or low values (e.g., modeling for supporting resources management). Published by Elsevier B.V.

1. Introduction Numerical models have been widely used in environmental science for understanding the natural processes, predicting impacts of global changes, and decision making for the sustainable management of resources. As knowledge of physical processes grows, models become more sophisticated and more parameters may be introduced (Beck, 1999; Brun et al., 2001; Legates and McCabe, 1999). We can see examples of the continuous developments of process-based models such as Soil and Water Assessment Tool (SWAT) (Arnold et al., 2012; Arnold et al., 1998) in hydrology and Erosion Deposition Carbon Model (EDCM) (Liu et al., 2003), a modified version of CENTURY (Parton et al., 1994), in ecology. These mathematical models include some parameters that need to be calibrated through an optimization procedure, which is to sample the parameter values from the allowable ranges until the value of the objective function (i.e., a function of differences between observations and simulations) is minimized or maximized (Diskin and Simon, 1977; Legates and McCabe, 1999; Nash and Sutcliffe, 1970). From a literature review, a number of objective functions were used for model calibration in hydrology such as mean squared error, absolute mean/maximum error, residual bias, and Nash objective function (Boyle et al., 2000; Diskin and Simon, 1977; Gupta et al., 1998; Servat and Dezetter, 1991; Yapo et al., 1998). However, the sum of square errors (SSR) is the most commonly used objective function for a variety of ⁎ Corresponding authors. E-mail addresses: [email protected], [email protected] (Y. Wu), [email protected] (S. Liu).

http://dx.doi.org/10.1016/j.ecoinf.2014.08.002 1574-9541/Published by Elsevier B.V.

optimization processes even in recent years (Confesor and Whittaker, 2007; Diskin and Simon, 1977; Gupta et al., 1998; Van Liew et al., 2005; Zhang et al., 2009). We also observe use of SSR in some popular optimization procedures such as the SWAT Auto-calibration Tool (Green and van Griensven, 2008; van Griensven, 2006; van Griensven et al., 2006), SWAT Calibration and Uncertainty Program (SWAT-CUP) (Abbaspour, 2012), the Flexible Model Environment (FME) R package (Soetaert and Petzoldt, 2010), and reservoir operation optimizations (Jothiprakash and Shanthi, 2006; Momtahen and Dariane, 2007; Raman and Chandramouli, 1996; Reddy and Kumar, 2006). In evaluating model performances mathematically, studies have illustrated that the correlation-based measures characterized by ‘square error’ such as square correlation coefficient (r2) and Nash–Sutcliffe Efficiency (NSE) are oversensitive to extreme values (outliers) and insensitive to additive and proportional differences between observations and simulations (Legates and Davis, 1997; Legates and McCabe, 1999; Moore, 1991). Legates and McCabe (1999) proposed a modified NSE (mNSE), which uses the ‘absolute error’ to replace the ‘square error’ in the original NSE calculation to evaluate the goodness-of-fit of hydrological models. For a similar purpose (avoiding oversensitivity to extreme values), Krause et al. (2005) revised the NSE based on relative deviations (i.e., replacing the ‘square error’ by ‘square relative deviation’). Using multiple examples, they concluded that both mNSE and rNSE can suppress the oversensitivity to peak values, and the latter is more sensitive to the low values (Krause et al., 2005). As stimulated by the above findings, we can infer that the widelyused objective function, SSR, also emphasizes the extreme values of a

108

Y. Wu, S. Liu / Ecological Informatics 24 (2014) 107–111

set of observation data and neglects the low values during model calibration because squaring calculation usually means a relatively larger weight for peak or higher values. Thus, we hypothesize that using the sum of absolute errors (SAR) and the sum of absolute relative deviations (SARD) may be better than SSR and the sum of square relative deviations (SSRD), respectively, for an environmental model calibration without emphasizing either high or low values. The objective of this study is to test this hypothesis by implementing model calibrations using these four different objective functions and evaluating the corresponding model performances. For this purpose, we used two large complex models in different disciplines: the widely-used hydrological model —SWAT—with a case study of monthly streamflow calibration in a headwater area of the East River Basin in South China and the wellestablished biogeochemical model—EDCM—with a case study of monthly gross primary production (GPP) calibration at a forest flux tower site in the eastern United States.

holding the others constant (such as optimization algorithm, input data, and calibration time period) during model calibration. This kind of scenario setting was the same for both hydrological modeling with SWAT and biogeochemical modeling with EDCM. 2.4. Criteria to assess model performance To assess different objective functions for model calibration, it is important to select a uniform and widely-accepted set of evaluation criteria. Because of the drawbacks (e.g., oversensitivity to extreme values) of correlation-based measures (e.g., NSE and r2) (see Introduction), the use of mNSE and mean absolute error (MAE) terms for overall assessment was recommended (Krause et al., 2005; Legates and McCabe, 1999). In this study, we adopted these two terms as the primary criteria to evaluate the model performances, although the other commonly-used terms NSE, r2, and RMSE are also reported for reference. The mathematical expressions of these five terms can be found in Table A.1 in Appendix A.

2. Materials and methods 3. Case studies 2.1. A hydrological model and a biogeochemical model The hydrological model, SWAT was developed by the U.S. Department of Agriculture (USDA) Agricultural Research Service for exploring the effects of climate and land cover changes on water, sediment, and agricultural chemical yields (Arnold et al., 1998). This physicallybased, watershed-scale, continuous model can simulate the hydrological cycle, cycles of plant growth, the transportation of sediment, and agricultural chemical yields on a daily time step (Arnold et al., 1998, 2012; Neitsch et al., 2005). The latest version, SWAT2012, was used in the current study. The biogeochemical model, EDCM (Liu et al., 2003), is a modified version of CENTURY (version IV) (Parton et al., 1994). EDCM uses up to 10 soil layers to simulate the soil organic carbon (SOC) dynamics in the whole soil profile instead of the one single top-layer structure of CENTURY. EDCM can dynamically keep track of the evolution of the soil profile and carbon storage as influenced by both soil erosion and deposition (Liu et al., 2003). This process-based biogeochemical model is used to simulate carbon and nitrogen cycles in diverse ecosystems at a monthly time step (Liu et al., 2003; Tan et al., 2009). In particular, was used to evaluate carbon dynamics across the entire conterminous United States (Liu et al., 2014; Liu et al., 2012b; Zhu, 2011; Zhu et al., 2010).

We used two case studies to illustrate the performances of objective functions during model calibrations on monthly streamflow and Gross Primary Production (GPP)—the two primary variables in hydrology and ecology, respectively. 3.1. Study area and model setup for hydrological modeling To drive SWAT for the hydrological modeling, we used the headwater area of the East River Basin (i.e., the Xunwu River) in South China as the case study, focusing on streamflow calibration, a common concern in hydrology. The Lizhangfeng flow gaging station has a drainage area of 1400 km2, and average annual precipitation is about 1648 mm in this area. The sources of input data (e.g., climate, topography, soil, and land use) are the same with what we used in previous studies where details can be found (Chen and Wu, 2012; Wu and Chen, 2013). In the current study, the SWAT setup with discretization resulted in the delineation of 11 subbasins and 67 Hydrological Response Units (HRUs) for the specific area. The calibration procedure was conducted using R-SWAT-FME with 5 years (1977–1981) of observed monthly streamflow at Lizhangfeng, and six streamflow-related parameters were selected in this study (see Table 1). 3.2. Study site and model setup for biogeochemical modeling

2.2. Modification of the modeling frameworks To implement the model calibration procedure for SWAT and EDCM, we used the developed R-SWAT-FME (Wu and Liu, 2012, 2014) and REDCM-FME (or EDCM-Auto) (Liu et al., 2012a; Wu et al., 2014), respectively. The two frameworks were developed to provide a variety of functionalities (e.g., parameter identifiability, optimization, and sensitivity and uncertainty analysis) for the two models (SWAT and EDCM), respectively. For the function of parameter optimization, the pseudo-random search algorithm (PseudoOptim) of Price (Price, 1977; Soetaert and Herman, 2009) included in FME was used in the current study, which was successfully tested for SWAT and EDCM calibrations. Because the original FME package uses SSR only as the objective function to compute model cost (Soetaert and Petzoldt, 2010), we modified the related function (modCost) to introduce the other three objective functions (i.e., SAR, SSRD, and SARD) we proposed as alternatives. The corresponding mathematical expressions of the four objective functions are listed in Table A.1 in Appendix A. 2.3. Scenarios for comparing objective functions For comparing the four objective functions, we set four scenarios with one objective function being assigned for each scenario while

For biogeochemical modeling with EDCM, we used a forest flux tower site—the Harvard Forest Environmental Monitoring Site, near Petersham, Massachusetts, in the United States (Curtis et al., 2002; Goulden et al., 1996)—with a focus on calibrating GPP, an elementary term in the carbon cycle. Soil texture data from the Ameriflux website indicate a soil composition of 66% sand, 29% silt, and 5% clay, with a bulk density of 0.9 g/cm3. Monthly precipitation and air temperature data were from the Parameter-elevation Regressions on Independent Slopes Model (PRISM) Climate Group (2012). Other model input data (e.g., soil organic carbon) were from the national data layers for the conterminous United States of the Land Carbon project (Zhu and Reed, 2012, in press). The derived GPP data obtained from the Ameriflux website were used as the observations during the 5-year (2001–2005) model calibration using R-EDCM-FME, and four parameters were involved in this procedure as listed in Table 1. 4. Results Using the R-SWAT-FME framework, we derived a set of optimal parameter values for each objective function in streamflow calibration (with the first case study of hydrological modeling). As listed in Table 1, the four sets of parameter values are quite different under the

Y. Wu, S. Liu / Ecological Informatics 24 (2014) 107–111

109

Table 1 Calibrated parameter values for SWAT and EDCM using the pseudo-random search algorithm (PseudoOptim) of the Flexible Modeling Environment (FME) with four different objective functions. Model

SWAT

EDCM

a b c d e

Parameter

CN2 SURLAG ESCO ALPHA_BF SOL_AWC SOL_K PRDX LEAFDR9 PPDF1 TEFF3

Description

Range

SCS curve number for moisture condition II Surface runoff lag coefficient Soil evaporation compensation factor Baseflow alpha factor (day) Soil available water capacity Saturated hydraulic conductivity (mm/h) Potential gross production (g C/m2) Leaf death rate in September Optimal temperature for production (°C) Temperature effect on soil decomposition rate (exponent value)

Calibrated value/change with different objective functions

e

−5%–+5% 0.1–3 0.1–1 0.001–0.06 0.1–0.4 5–40 800–1500 0.3–0.8 28–30 0.01–0.3

SSRa

SARb

SSRDc

SARDd

0.2% 0.88 0.28 0.001 0.30 5.0 1150 0.49 28 0.03

−4.8% 0.15 0.88 0.006 0.34 12.6 1195 0.59 28 0.29

3.7% 2.89 0.26 0.001 0.36 5.7 1407 0.8 28.5 0.12

−5.0% 0.1 0.57 0.005 0.38 16.4 1385 0.7 28.6 0.22

SSR: sum of squared error. SAR: sum of absolute error. SSRD: sum of squared relative deviation. SARD: sum of absolute relative deviation. CN2 changes relative to the default values.

four scenarios (objective functions). However, the evaluation of streamflow simulations indicates that the model performances for the four scenarios were all acceptable, with NSE N 0.85, r2 N0.87, and mNSE N 0.67 (Table 2). In other words, the four sets of simulated streamflow were quite close and matched well with the observation. From the numerical scales, we can still identify that SAR and SARD (using absolute error) were better than SSR and SSRD (using square error), respectively, in terms of mNSE and MAE. Among the four scenarios, SAR behaved the best among the four scenarios, but SSR demonstrated the best performance if we use measures of NSE and RMSE (which were not recommended for model performance evaluation) (Table 2). Although r2 measures showed that SAR was the best, the value for SAR was marginally higher than SSR. Overall, we can say using SAR resulted in the best model performance in streamflow simulations based on the more reasonable and acceptable measures of mNSE and MAE (see Introduction and Section 2.4). Similarly, the second case study—biogeochemical modeling—also demonstrated that although the GPP simulations were close and agreed well with the observations under any of the four scenarios, with NSE Table 2 Evaluation of monthly streamflow and gross primary production (GPP) simulations by SWAT and EDCM, respectively, during model calibrations. Model

Objective function

SWATg

SSR SAR SSRD SARD SSR SAR SSRD SARD

EDCMh

Meana 37.32

238.7

mNSEb

MAEc

NSEd

r2e

RMSEf

0.704 0.741 0.672 0.736 0.808 0.817 0.718 0.757

6.714 5.863 7.431 5.99 42.105 40.005 61.797 53.157

0.884 0.873 0.855 0.875 0.945 0.937 0.841 0.881

0.890 0.895 0.878 0.885 0.945 0.939 0.874 0.900

9.716 10.185 10.874 10.1 56.598 60.393 96.007 83.164

Note: Bold numbers indicate the corresponding objective function which resulted in the best model performance for a specific evaluation term, but mNSE and MAE were regarded as the reasonable assessing criteria. SSR, SAR, SSRD, and SARD have the same meaning as in Table 1. a Mean: mean value of observation, which has units of m3/s for streamflow and g C/m2/ month for GPP. b mNSE: modified Nash–Sutcliffe efficiency. c MAE: mean absolute error, which has units of m3/s for streamflow and g C/m2/month for GPP. d NSE: Nash–Sutcliffe efficiency. e r2 is squared correlation coefficient. f RMSE: root mean square error, which has units of m3/s for streamflow and g C/m2/ month for GPP. g The five-year calibration period for SWAT is 1977 to 1981. h The five-year calibration period for EDCM is 2001 to 2005.

N0.84, r2 N 0.87, and mNSE N 0.71 (Table 2), using SAR would have the best model performance in terms of mNSE and MAE. 5. Discussion As described in the previous section, different sets of parameter values may result in similar model behavior—streamflow and GPP simulations in the two case studies. This non-uniqueness is common and is an inherent property of inverse modeling (Abbaspour et al., 1997; Beven and Binley, 1992; Duan et al., 1992; Gupta et al., 1998). However, this kind of equifinality is beyond the scope of this study, which focused on the comparison of the different objective functions in model calibration (i.e., which one may lead to the least model error and best model performance for a given modeling case). Although squaring error to avoid the canceling of errors with opposite signs may be easier to manipulate mathematically than using absolute values (Legates and McCabe, 1999), it forces an arbitrarily greater influence on larger values (Krause et al., 2005; Legates and Davis, 1997; Legates and McCabe, 1999; Moore, 1991). Thus, using ‘square error’ for calculating objective function (e.g., SSR) can make the optimization procedure highly sensitive to higher values. Using RMSE (with squaring error) as an objective function was reported with such a result (Boyle et al., 2000). Our case studies with hydrological and biogeochemical model calibrations indicated that SSR did not result in the least MAE and the highest mNSE, but SAR did. We acknowledge that different objective functions may give more or less emphasis to different parts of observation data (Diskin and Simon, 1977). For example, extreme values are important for flood modeling in which square error or even the fourth power of the error (square twice) could be better than the absolute error; low values are the major concern in dry season flow modeling in which relative deviation may be better (Krause et al., 2005); and no emphasis on any part of the data is needed for applications supporting long-term resource management such as the current topics of climate and land use changes on water quantity and quality and carbon sequestration where absolute error appears to be more reasonable. Therefore, we are not suggesting that SSR should be totally replaced by SAR in calculating objective functions. With two representative cases in hydrology and biogeochemistry, however, we believe that SSR may not be suitable as a dominant objective function as we have seen in optimization applications (see Introduction for its wide use) because it emphasizes high values and weakens low values. Instead, SAR can be superior in this aspect because it places an equal weight to each observation data. This suggestion can also be applied to other types of objective functions involving absolute or square errors.

110

Y. Wu, S. Liu / Ecological Informatics 24 (2014) 107–111

In a subjective view, if multiple objective functions are available for a model calibration procedure, SAR (or other SAR-related formulae) may still deserve a default choice because users who are not concerned about either high or low values, may not care about the objective function when they use a model inversion procedure, especially considering that most users may not (or do not need to) scrutinize the details of a published procedure (e.g., FME and SWAT-CUP). Conversely, users who are inclined to emphasize extreme values (e.g., flood modeling) will usually intend to incorporate selection of an objective function, which can force them to investigate the procedure they are planning to use and modify it as needed (Kang, 2011; Krause et al., 2005; Legates and McCabe, 1999). Using the popular FME R package as an example, SSR is the only available objective function, and few users would consider modifying it by scrutinizing into the code. However, our case studies (using R-SWAT-FME and R-EDCM-FME) suggest that SAR may be a more appropriate choice than SSR (or at least a default one) for this package. 6. Conclusions In this study, we examined four potential objective functions (SSR, SAR, SSRD, and SARD) in model calibration followed by model performance evaluations using two case studies: hydrological modeling with streamflow calibration and biogeochemical modeling with GPP calibration. The results demonstrate that ‘absolute error’ (SAR and SARD) are superior to ‘square error’ (SSR and SSRD) in calculating objective function for model calibrations, and SAR behaved the best, resulting in the least error (MAE) and the highest model efficiency (mNSE). As a result of this study, we believe that SSR might be overly used in optimization applications. If a dominant or default objective function is needed, SAR might be a reasonable choice. Acknowledgments This study was funded by the Land Carbon Project (GEMS Modeling), U.S. Carbon Trends, and U.S. Geological Survey (USGS) Geographic Analysis and Monitoring (GAM) Program. The work was performed under USGS contract G08PC91508. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government. We thank Shengli Huang (ASRC Federal, contractor to USGS EROS Center) for performing internal review and Sandra Cooper (USGS) for further editing. We are also grateful to the anonymous reviewer for the constructive comments and suggestions. Appendix A Table A.1 Primary mathematical expressions involved in this study. Name

Description

SSR SAR SSRD

Sum of square errors Sum of absolute errors Sum of square relative deviations

SARD

Sum of absolute relative deviations Nash–Sutcliffe efficiency

NSE

Equation

Number 2

U = ∑ (S − O) U = ∑ |S − O|
 S−O 2 U¼∑ O   S−O  U ¼ ∑ O 

(A.1) (A.2) (A.3)

2 NSE¼1−∑ðS−OÞ  2

(A.5)

∑ O−O     2 ∑ O−O  S−S 2 r ¼  2  2 ∑ O−O  ∑ S−S rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ∑ðS−OÞ2 RMSE ¼ n

(A.4)

r2

Square correlation coefficient

RMSE

Root mean square error

mNSE

Modified NSE

mNSE¼1−∑jS−Oj

(A.8)

MAE

Mean absolute error

MAE ¼ 1n ∑jS−Oj

(A.9)

  ∑O−O

(A.6)

(A.7)

Note: U is the objective function to be minimized; O and S are observations and simulations, respectively; O and S are mean values of observations and simulations, respectively; n is the number of the data points. The first four items are for objective functions, and the last five are for evaluating model performance.

References Abbaspour, K.C., 2012. SWAT Calibration and Uncertainty Programs 2012. 103. Swiss Federal Institute of Aquatic Science and Technology, Eawag, Duebendorf, Switzerland. Abbaspour, K.C., vanGenuchten, M.T., Schulin, R., Schlappi, E., 1997. A sequential uncertainty domain inverse procedure for estimating subsurface flow and transport parameters. Water Resour. Res. 33, 1879–1892. Arnold, J.G., Srinivasan, R., Muttiah, R.S., Williams, J.R., 1998. Large area hydrologic modeling and assessment — part 1: model development. J. Am. Water Resour. Assoc. 34, 73–89. Arnold, J.G., Kiniry, J.R., Srinivasan, R., Williams, J.R., Haney, E.B., Neitsch, S.L., 2012. Soil and Water Assessment Tool Input/Output Documentation. Texas Water Resources Institute, Grassland, Soil and Research Service, Temple, TX. Beck, M.B., 1999. Coping with ever larger problems, models, and data bases. Water Sci. Technol. 39, 1–11. Beven, K., Binley, A., 1992. The future of distributed models — model calibration and uncertainty prediction. Hydrol. Process. 6, 279–298. Boyle, D.P., Gupta, H.V., Sorooshian, S., 2000. Toward improved calibration of hydrologic models: combining the strengths of manual and automatic methods. Water Resour. Res. 36, 3663–3674. Brun, R., Reichert, P., Kunsch, H.R., 2001. Practical identifiability analysis of large environmental simulation models. Water Resour. Res. 37, 1015–1030. Chen, J., Wu, Y., 2012. Advancing representation of hydrologic processes in the Soil and Water Assessment Tool (SWAT) through integration of the TOPographic MODEL (TOPMODEL) features. J. Hydrol. 420–421, 319–328. Confesor Jr., R.B., Whittaker, G.W., 2007. Automatic calibration of hydrologic models with multi-objective evolutionary algorithm and Pareto optimization. J. Am. Water Resour. Assoc. 43, 981–989. Curtis, P.S., Hanson, P.J., Bolstad, P., Barford, C., Randolph, J.C., Schmid, H.P., et al., 2002. Biometric and eddy-covariance based estimates of annual carbon storage in five eastern North American deciduous forests. Agric. For. Meteorol. 113, 3–19. Diskin, M.H., Simon, E., 1977. A procedure for the selection of objective function for hydrologic simulation models. J. Hydrol. 34, 129–149. Duan, Q.Y., Sorooshian, S., Gupta, V., 1992. Effective and efficient global optimization for conceptual rainfall–runoff models. Water Resour. Res. 28, 1015–1031. Goulden, M.L., Munger, J.W., Fan, S.M., Daube, B.C., Wofsy, S.C., 1996. Measurements of carbon sequestration by long-term eddy covariance: methods and a critical evaluation of accuracy. Glob. Chang. Biol. 2, 169–182. Green, C.H., van Griensven, A., 2008. Autocalibration in hydrologic modeling: using SWAT2005 in small-scale watersheds. Environ. Model Softw. 23, 422–434. Gupta, H.V., Sorooshian, S., Yapo, P.O., 1998. Toward improved calibration of hydrologic models: multiple and noncommensurable measures of information. Water Resour. Res. 34, 751–763. Jothiprakash, V., Shanthi, G., 2006. Single reservoir operating policies using genetic algorithm. Water Resour. Manag. 20, 917–929. Kang, H., 2011. Evaluation of SWAT auto-calibration using diverse efficiency criteria. 2011 International SWAT Conference. 34 (http://swat.tamu.edu/media/39113/kang.pdf). Krause, P., Boyle, D.P., Base, F., 2005. Comparison of different efficiency criteria for hydrological model assessment. Adv. Geosci. 5, 89–97. Legates, D.R., Davis, R.E., 1997. The continuing search for an anthropogenic climate change signal: limitations of correlation-based approaches. Geophys. Res. Lett. 24, 2319–2322. Legates, D.R., McCabe, G.J., 1999. Evaluating the use of “goodness-of-fit” measures in hydrologic and hydroclimatic model validation. Water Resour. Res. 35, 233–241. Liu, S.G., Bliss, N., Sundquist, E., Huntington, T.G., 2003. Modeling carbon dynamics in vegetation and soil under the impact of soil erosion and deposition. Glob. Biogeochem. Cycles 17. Liu, S., Liu, J., Young, C., Werner, J.M., Wu, Y., Li, Z., et al., 2012a. Baseline Carbon Storage, Carbon Sequestration, and Greenhouse-Gas Fluxes in Terrestrial Ecosystems of the Western United States. . Chapter 5 In: Zhu, Z., Reed, B. (Eds.), U.S. Geological Survey, Reston, Virginia, pp. 45–63 (Available online at http://pubs.usgs.gov/pp/1797/pdf/ pp1797_Chapter5.pdf). Liu, S., Wu, Y., Young, C., Dahal, D., Werner, J.M., Liu, J., et al., 2012b. Projected Future Carbon Storage and Greenhouse-Gas Fluxes of Terrestrial Ecosystems in the Western United States. Chapter 9 In: Zhu, Z., Reed, B. (Eds.), U.S. Geological Survey, Reston, Virginia, pp. 109–124 (Available online at http://pubs.usgs.gov/pp/1797/pdf/ pp1797_Chapter9.pdf). Liu, S., Liu, J., Wu, Y., Young, C., Werner, J., Dahal, D., et al., 2014. Baseline and Projected Future Carbon Storage, Carbon Sequestration, and Greenhouse-gas Fluxes in Terrestrial Ecosystems of the Eastern United States. Chapter 7 U.S. Geological Survey, Reston, Virginia, pp. 115–156, (Available online at http://pubs.usgs.gov/pp/1804/pdf/ pp1804_chapter7.pdf). Momtahen, S., Dariane, A.B., 2007. Direct search approaches using genetic algorithms for optimization of water reservoir operating policies. J. Water Resour. Plann. Manage. 133, 202–209. Moore, D.S., 1991. Statistics: Concepts and Controversies. W.H. Freeman, New York, (439 pp.).

Y. Wu, S. Liu / Ecological Informatics 24 (2014) 107–111 Nash, J.E., Sutcliffe, J.V., 1970. River flow forecasting through conceptual models. Part I. A discussion of principles. J. Hydrol. 10, 282–290. Neitsch, S.L., Arnold, J.G., Kiniry, J.R., Williams, J.R., King, K.W., 2005. Soil and Water Assessment Tool Theoretical Documentation. Grassland, Soil and Research Service, Temple, TX. Parton, W.J., Ojima, D.S., Schimel, D.S., 1994. Environmental change in grasslands: assessment using models. Clim. Chang. 28, 111–141. Price, W.L., 1977. A controlled random search procedure for global optimization. Comput. J. 20. PRISM Climate Group, 2012. Parameter-elevation Regressions on Independent Slopes Model (PRISM) data: PRISM Climate Group Dataset. (http://www.prism. oregonstate.edu/. Accessed October 23, 2012) USDA Risk Management Agency. Raman, H., Chandramouli, V., 1996. Deriving a general operating policy for reservoirs using neural network. J. Water Resour. Plann. Manage. 122, 342–347. Reddy, M.J., Kumar, D.N., 2006. Optimal reservoir operation using multi-objective evolutionary algorithm. Water Resour. Manag. 20, 861–878. Servat, E., Dezetter, A., 1991. Selection of calibration objective functions in the context of rainfall–runoff modeling in a Sudanese savannah area. Hydrol. Sci. J. 36, 307–330. Soetaert, K., Herman, P.M.J., 2009. A Practical Guide to Ecological Modeling. Springer-Verlag, New York. Soetaert, K., Petzoldt, T., 2010. Inverse modelling, sensitivity and Monte Carlo analysis in R using Package FME. J. Stat. Softw. 33, 1–28. Tan, Z., Tieszen, L.L., Tachie-Obeng, E., Liu, S., Dieye, A.M., 2009. Historical and simulated ecosystem carbon dynamics in Ghana: land use, management, and climate. Biogeosciences 6, 45–58. van Griensven, A., 2006. Sensitivity, Auto-calibration, Uncertainty and Model Evaluation in SWAT2005, p. 48, (Available online at http://biomath.ugent.be/~ann/swat_ manuals/SWAT2005_manual_sens_cal_unc.pdf (Accessed March 6, 2012)). van Griensven, A., Meixner, T., Grunwald, S., Bishop, T., Diluzio, A., Srinivasan, R., 2006. A global sensitivity analysis tool for the parameters of multi-variable catchment models. J. Hydrol. 324, 10–23. Van Liew, M.W., Arnold, J.G., Bosch, D.D., 2005. Problems and potential of autocalibrating a hydrologic model. Trans. ASAE 48, 1025–1040.

111

Wu, Y., Chen, J., 2013. Analyzing the water budget and hydrological characteristics and responses to land use in a monsoonal climate river basin in South China. Environ. Manag. 51, 1174–1186. Wu, Y., Liu, S., 2012. Automating calibration, sensitivity and uncertainty analysis of complex models using the R package Flexible Modeling Environment (FME): SWAT as an example. Environ. Model Softw. 31, 99–109. Wu, Y., Liu, S., 2014. Improvement of the R-SWAT-FME framework to support multiple variables and multi-objective functions. Sci. Total Environ. 466–467, 455–466. Wu, Y., Liu, S., Li, Z., Dahal, D., Young, C.J., Schmidt, G.L., et al., 2014. Development of a generic auto-calibration package for regional ecological modeling and application in the Central Plains of the United States. Ecol. Inform. 19, 35–46. Yapo, P.O., Gupta, H.V., Sorooshian, S., 1998. Multi-objective global optimization for hydrologic models. J. Hydrol. 204, 83–97. Zhang, X.S., Srinivasan, R., Bosch, D., 2009. Calibration and uncertainty analysis of the SWAT model using genetic algorithms and Bayesian model averaging. J. Hydrol. 374, 307–317. Zhu, Z., 2011. Baseline and Projected Future Carbon Storage and Greenhouse-gas Fluxes in the Great Plains Region of the United States. U.S. Geological Survey, Reston, Virginia, p. 28, (Available online at http://pubs.usgs.gov/pp/1787/p1787.pdf). Zhu, Z., Reed, B., 2012. Baseline and Projected Future Carbon Storage and Greenhouse-gas Fluxes in Ecosystems of the Western United States. U.S. Geological Survey, Reston, Virginia, pp. 45–63, (Available online at http://pubs.usgs.gov/pp/1797/pdf/pp1797.pdf). Zhu, Z., Reed, B., 2014. Baseline and projected future carbon storage and greenhouse-gas fluxes in ecosystems of the eastern United States. U.S. Geological Survey Professional Paper 1804. U.S. Geological Survey, Reston, Virginia pp. 204 (Available online at http://pubs.usgs.gov/pp/1804). Zhu, Z., Bergamaschi, B., Bernknopf, R., Clow, D., Dye, D., Faulkner, S., et al., 2010. A Method for Assessing Carbon Stocks, Carbon Sequestration, and Greenhouse-gas Fluxes in Ecosystems of the United States Under Present Conditions and Future Scenarios. Reston, Virginia. Available online at http://pubs.usgs.gov/sir/2010/5233/.