MODELING Evaluating CROPGRO-Soybean Performance for Use in Climate Impact Studies Gregory J. Carbone,* Linda 0. Mearns, Theodoros Mavromatis, E. John Sadler, and David Stooksbury coefficients, soil wilting point, field capacity, and saturation levels. Most studies estimating how climate change influences agriculture are conducted at regional or continental scales. At these scales, practical considerations limit generation of the spatially detailed weather, cultivar, soil, and management inputs demanded by many crop models. Therefore, researchers often must characterize regional conditions without access to detailed physical samples (e.g., Curry et al., 1995; Alexandrov and Hoogenboom, 2000). There is a need to evaluate crop models using a methodology similar to that used in many climate impact studies (e.g., Mearns et al., 1999, 2001; Brown et al., 2000; Easterling et al., 2001). Yield trials from state agricultural experiment stations provide an appropriate data source to examine general model performance during many consecutive years (Mavromatis et al., 2001,2002). While data limitations preclude intensive model evaluation at the experiment stations, the relatively long time series pf yield is particularly relevant to most research on agricultural impacts where climate change scenarios are developed by adjusting observed meteorological time series and preserving interannual variability (Acock and Acock, 1991). Evaluation of the magnitude and pattern of deviations between simulated and observed yield for multiple years and stations can measure, under relatively controlled conditions, how uncertainties inherent in model inputs affect simulated yield estimates (Bouman, 1994; Moen et al., 1994; Nonhebel, 1994; Hansen and Jones, 2000). Moreover, crop model simulation for multiple years allows evaluation of interannual variability, an important quality for climate impact research (Aggarwal, 1995; Mearns et al., 1996, 1997; Kiniry et al., 1997).
ABSTRACT Researchers frequently use crop simulation models to estimate the impacts of climate change on agricultural production. While most models used for this purpose have been validated thoroughly at the research plot level, few studies have evaluated them for multiple years and sites and with inputs commonly used in climate impact studies. Here, we examine how well CROPGRO-Soybean performs across space and time using cultivar coefficients provided with the model, estimates of solar radiation, and soil inputs that were estimated from readily available soil surveys. Modeled yield for three maturity groups was compared with that observed from 8 to 23 yr at eight agricultural experiment stations in the southeastern United States. The model was evaluated tvith respect to its ability to replicate the mean and standard deviation of observed yield. The mean squared deviation (MSD), weighted according to the number of years at each station, was 0.42 (Mg ha'9)1. The model simulated mean yield and the magnitude of interannual yield variability very well. The component of MSD related to the pattern of interannual variability contributed most to MSD. Our results support the use of crop models in studies that require accurate simulation of the temporal mean and variance of yields.
PROCESS-ORIENTED
CROP MODELS have been used exltensively in climate impact studies (Rosenzweig and Iglesias, 1994; Hoogenboom et al., 1995; Haskett et al., 1997; Mearns et al., 1999; Alexandrov and Hoogenboom, 2000; Guerefia et al., 2001) because they simulate crop response to changing climatic conditions. This confidence results from model validation at the scale of research plots. For example, the model used in this study-CROPGRO-Soybean (Hoogenboom et al., 1994; Boote et al., 1998)-has been shown to accurately predict reproductive development and yield (Brisson et al., 1989; Colson et al., 1995; Sau et al., 1999) at various field locations. Because researchers often use these models to investigate climate impacts, the models also should be evaluated in ways that supplement this field-level validation and more closely resemble the means by which inputs are derived for impact studies. This includes simulation during multiple years, at different locations, and with inputs that are not measured directly in the field, but estimated from other published or unpublished data sources. Such inputs include solar radiation, cultivar
MATERIALS AND METHODS CROPGRO-Soybean and Data Inputs CROPGRO-Soybean v. 3.5 is a process-oriented model that simulates C, water, and N balances for the soybean [Glycine max (L.) Merr.] plant and soil (Hoogenboom et al., 1994; Boote et al., 1998). Model equations express relationships between plant processes (including phenological development, photosynthesis, respiration, biomass growth and partitioning, and soil-plant water balance) and daily temperature, photoperiod, insolation, and water or N stress. Growth is integrated at a daily time step, ultimately predicting biomass,
G.J. Carbone, Dep. of Geogr., Univ. of South Carolina, Columbia, SC 29208; L.O. Mearns, Natl. Cent. for Atmos. Res., P.O. Box 3000, Boulder, CO 80303; T. Mavromatis, Dep. of Agric. and Biol. Eng., Univ. of Florida, Gainesville, FL 32611; E.J. Sadler, USDA-ARS, Coastal Plains Soil, Water, and Plant Res. Cent., 2611 W. Lucas St., Florence, SC 29501; and D. Stooksbury, Dep. of Biol. and Agric. Eng., Univ. of Georgia, Athens, GA 30602. Received 17 Sept. 2001. *Corresponding author (
[email protected]).
Abbreviations: LCS, lack of correlation weighted by the standard deviations; MSD, mean squared deviation; SB, squared bias; SD, standard deviation; SDSD, squared difference between standard deviations.
Published in Agron. J. 95:537-544 (2003).
537
538
AGRONOMY JOURNAL, VOL. 95, MAY-JUNE 2003
leaf area, seed dry weight, and yield. Cultivar coefficients determine duration within growth stages, response to daylength, maximum growth rates, and seed characteristics. Like many similar crop models, CROPGRO-Soybean does not explicitly treat weeds, diseases, pests, wind damage, or other extreme weather events although efforts have been made to address some of these issues (Batchelor et al., 1993). The model requires daily weather inputs, soil parameters at various profile depths, and cultural or management details. In this study, direct field measurements from eight agricultural experiment stations in Alabama, Georgia, Louisiana, and South Carolina, USA (Fig. 1), were used as inputs whenever possible. For example, daily maximum and minimum air temperatures and precipitation were obtained from the cooperative weather stations associated with each experiment station. In contrast, solar radiation was measured at only one station (Tifton, GA) and had to be generated stochastically with WXGEN (Richardson and Nicks, 1990) elsewhere. Soil type was published for each experiment station, and descriptions of each soil used in this study are available from USDA (Soil Survey Div., 2001). Detailed texture characteristics and organic matter and compaction data for each soil were derived from the STATSGO soils database (USDA-NRCS, 2001). These characteristics were used to estimate saturation point and drained upper and lower limit (Baumer and Rice, 1988; Nat. Resour. Conserv. Serv., 1988; Baumer et al., 1994). The method estimates volumetric soil water content for a range of matric potentials and then fits a soil water retention curve using the RETC program (van Genuchten and Nielsen, 1985) to describe hydraulic properties of unsaturated soils. While such estimation methods produce results that vary from field-measured values (Gijsman et al., 2002), they are used frequently because field-measured wilting point, field capacity, and saturation are not available across broad regions. We used agricultural experiment station performance reports for crop and management inputs, including row spacing, plant population, planting date, irrigation and fertilizer application, and cultivar. CROPGRO-Soybean v. 3.5 includes 15 cultivar coefficients that characterize phenology and vegetative and reproductive growth. Published values of these coeffi-
Fig. 1. Locations of eight agricultural experiment stations used in analysis.
cients are available for a variety of individual cultivars as well
as generic maturity groups, 000 to X (Tsuji et al., 1994). These latter cultivar coefficients were derived from experiments with several varieties belonging to each particular maturity group. We used published cultivar coefficients for two specific cultivars-Forrest (maturity group V) and Ransom (maturity group VII)-at the three Alabama stations where these varieties were consistently grown during a 20-yr period. At the
other stations, we used cultivar coefficients for generic maturity groups V, VI, and VII. We did not tune the coefficients for local conditions using observed data. While this could affect model performance, our goal was to measure how weli the model performs using data typically available for a climate impact study. For all simulations, the CROPGRO model was run with water balance, soil N balance, N fixation, and plant N balance options turned on. We ran the crop model for 4 yr before the first growing season to estimate initial soil water and N amounts. We ran the model in sequence mode through consecutive years to preserve soil moisture values from one season to the next. Simulation length varied according to data availability at each station (Table 1). Evaluating Model Performance Agricultural experiment station performance reports also provided observed crop yield data. We examined model performance using MSD, the index of agreement (d), and coefficient of determination (r2) (Willmott, 1982). These measures were applied to individual stations but were also aggregated by maturity group to summarize model performance across space and time. We also employed the methods of Kobayashi and Salam (2000) to decompose MSD for individual stations. In their method, MSD consists of three parts: MSD = SB + SDSD + LCS. These three components characterize (i) model squared bias (SB), or the squared difference between average simulated and average observed yield; (ii) the magnitude of variability (SDSD), measured as the squared difference between simulated and observed standard deviations (SDs); and (iii) the pattern of variability between observed and simulated
539
CARBONE ET AL.: EVALUATING CROPGRO-SOYBEAN PERFORMANCE
Table 1. General simulation features at the eight agricultural experiment stations. The National Weather Service Cooperative Nehvork Station Number is provided.for each site. Cultivar or maturity group
Years
Soil type
Forrest Ransom Forrest
21 16 18
Ransom Forrest Ransom MG V MG VII MG VI MG VII MG VI MG VU
21 20 23 8 8 10 10 10 10
St. Joseph, LA (168163)
MG VI MG VII
9 9
Blackville, SC (380764)
MG V MG VI MG Vl
6 11 11
HartseDls fine sandy loam (fine-loamy, siliceous, thermic Typic Hapludults) Sumter clay (fine-silty, carbonatic, thermic Rendollic Eutrudepts); Houston day (very-fine, smectitic, thermic Oxyaquic Hapluderts); Vaiden day (very-fine, smectitic, thermic Aquic Dystruderts) Malbis fine sandy loam (fine-loamy, siliceous, subactive, thermic Plinthic Paleudults) Tifton sandy loam, Tifton sandy clay loam, Tifton loamy sand (fine-loamy, kaoGinitic, thermic Plinthic Kandiudults) Norwood silt loam (fine-silty, mixed, superactive, hyperthermic Fluventic Eutrudepts) Balwin silty clay loam (fine, smectitic, hyperthermic Chromic Vertic Epiaqualfs); Iberia silty clay loam (very-fine, smectitic, hyperthermic Typic Epiaquerts) Sharkey clay (very-fine, smectitic, thermic Chromic Epiaquerts); Commerce silt loam (fine-slty, mixed, superactive, nonacid, thermic Fluvaquentic Endoaquepts) Varina loamy sand (fine, kaolinitic, thermic Plinthic Paleudults)
Station Crossville, AL (017207) Marion Junction., AL (015121) Fairhope, AL (012813) Tifton, GA (irrigated) (098703) Alexandria, LA (165630) Jeanerette, LA (164674)
yield (LCS), which measures the lack of correlation weighted by the SDs of observed and simulated yield. These measures allowed us to examine the relative contribution to MSDs between modeled and observed yield (adjusted to account for moisture content). Our analysis included examination of overall model performance at individual stations and for specific varieties or maturity groups. When summarizing for a particular maturity group, we weighted results by the number of seasons of each contributing station. We compared differences between mean observed and mean simulated yield using t tests. The experiment station reports also provided qualitative records of seasonal conditions that help to describe disease and insect problems, lodging, and plant stress that results from nutrient deficiency, water stress, or extreme weather conditions.
RESULTS Maturity Group V The weighted mean simulated yield for maturity group V across all stations and years was 19% higher than mean observed yield. Simulated yield was significantly higher (a, = 0.05) overall and at two individual stations-Fairhope and Marion Junction. The SD was slightly higher for simulated yield than observed yield at each station while the coefficient of variation for simulated and observed yield was about the same (Table 2). The MSD ranged from 0.36 (Mg ha-') 2 at Tifton to 0.96 (Mg ha'1)2 at Marion Junction (Fig. 2). The LCS component was the largest fraction of MSD at
most stations, indicating that the model captured mean yield and the magnitude of interannual yield variation better than the pattern of interannual yield. Closer examination reveals that a few anomalous years disproportionately inflate the MSD. At Crossville, for example, heavy rains in 1972 delayed harvest until 16 Jan. 1973 (Thurlow, 1973). In 1975 and 1979, severe lodging re-
duced harvestable yield (Fig. 3a; Thurlow, 1976; Thurlow and Thomason, 1980). At Marion Junction, the model also captured mean yield and the magnitude of variability well but did not reproduce the pattern of interannual variability as well. The results at this station were strongly influenced by large overestimation during 1977, 1986, and 1987 (Fig. 3b) when severe Fe chlorosis reduced yield (Thurlow, 1978; Thurlow and Johnson, 1987,1988). Such nutrient deficiencies are not simulated by CROPGRO-Soybean; all factors not explicitly accounted for are considered nonlimiting. When these years are eliminated, MSD drops significantly (Table 3). Maturity Group VI The weighted mean simulated yield when all maturity group VI varieties were pooled was 6% higher than observed; this difference was not significant at the (x = 0.05 level. The difference between simulated and observed yield also was not significant at any individual
station (Table 4). The SD of simulated yield was higher than that observed in Blackville, but at other stations,
2 Table 2. Error statistics for maturity group V and Forrest cultivar (SD, standard deviation; CV, coefficient of variation; r , coefficient of determination; d, index of agreement).
Maturity group V N Mean simulated yield, Mg ha-' Mean observed yield, Mg ha-' SD simulated yield, Mg ha-' SD observed yield, Mg ha-' CV simulated yield CV observed yield r2 d
Fairhope
Crossville
Marion Junction
Blackville
Tifton
20 2.49* 2.10 0.96 0.85 0.38 0.40 0.74 0.77
21 2.24 2.00 0.86 0.64 0.38 0.32 0.42 0.77
18 1.62* 1.11 0.68 0.51 0.42 0.46 0.08 0.49
6 1.85 1.29 0.66 0.39 0.35 0.30 0.63 0.82
8 2.91 2.66 1.61 1.42 0.55 0.53 0.91 0.88
* Significantly different than observed yield (oc = 0.05).
540
AGRONOMY JOURNAL, VOL. 95, MAY-JUNE 2003
Components of Mean Squared Devia tion 2.0 Maturity Group V
2.00
LCS
. 1.5- _SDSD a)
-1.5 -
SB
~0
-
U) 1.0--0r
-1~i .0 -
i..
cu C/,
cu 0.5a) -
An_
-0.5
]
I
r_____1
-J
0) 0 N> CD X 0) 0) 0)
Marion Junction, AL, Forrest
5-
-5
* simulated
o observed
-4
0
3-
-3
(a cm
0 2-
0
0
0
-2
0 0 .
0
0 a
0
°
o
II I
01
* 0
Without 1977, 1986,1987 0.08 0.05 0.41 0.54
-3
0
0
Marion Junction (Forrest cultivar)
0.30 0.06 0.60 0.96
* 0
a
0
u-
SB SDSD LCS MSD
0 o 0 &
0-
All years
0
S
3-
2-
-4
0
.1i
IU
MSD and its components
o observed
0)
The weighted mean simulated yield for maturity group VII was about 13% higher than observed (Table 5). While this difference is significant at the 0.01 level, Fairhope was the only individual station with a significant difference (ot = 0.01) between observed and simulated yield. The SD of simulated yield was consistently higher than observed yield, but the coefficient of variation of simulated and observed yield was nearly
Crossville (Forrest cultivar)
-5
* simulated
-
Maturity Group VII
Table 3. The influence of anomalous years on mean squared deviation (MSD) and its components-squared bias (SB), lack of correlation weighted by the standard deviations (LCS), and squared difference between standard deviations standardized bias (SDSD).
Crossville, AL, Forrest
5-
I
I I I I I N1l C' I) lo 1C co 0 0 r'- C' C0r'- C'- C X'0 0) 0)0) )0 ) 0 0)0)
-1
0
0
*0
I I C I N C ts ) Co CD X U Um a co0 0)00))0))0))0))
I I I I e- UX 0 0 U(0) 0
0
0 I 00
N
Fig. 3. Time series of simulated and observed yield for Forrest cultivar at (a) Crossville, AL, and (b) Marion Junction, AL.
541
CARBONE ET AL.: EVALUATING CROPGRO-SOYBEAN PERFORMANCE
Table 4. Error statistics for maturity group VI (SD, standard deviation; CV, coefficient of variation; r2 , coefficient of determination; d, index of agreement). Maturity group VI N Mean simulated yield, Mg ha-' Mean observed yield, Mg ha' SD sinulated yield, Mg ha-' SD observed yield, Mg ha-' CV simulated yield CV observed yield r2 d
Blackville
Alexandria
Jeanereffe
St. Joseph
11 1.60 1.42 0.66 0.38 0.41 0.49 0.61 0.77
10 2.43 2.20 0.87 0.81 0.36 0.37 0.65 0.58
10 2.10 2.01 0.84 0.77 0.40 0.38 0.79 0.82
9 1.83 1.90 1.08 1.05 0.59 0.55 0.87 0.90
following components: SB = 0.09, SDSD = 0.03, and LCS = 0.35. The total MSD value should be viewed in the context of previous studies evaluating the performance of CROPGRO-Soybean. Jones and Ritchie (1990) reported relatively small model deviation from observed yield measurements [about 0.2 (Mg ha') 2 ] using fieldmeasured inputs. In their experiments evaluating model response to planting date, Egli and Bruening (1992) had an MSD value of about 0.25 (Mg ha-1)2 . They too used inputs measured directly in the field, such as solar radiation and soil water-holding characteristics. Mavromatis et al. (2001) reported an MSD value of 0.59 (Mg ha-')2 for the 'Stonewall', maturity group VII cultivar. They reduced the MSD for this cultivar to 0.163 (Mg ha-') 2 by systematically adjusting cultivar coefficients and soil parameters to minimize the squared difference between simulated and observed yield. In a subsequent paper, Mavromatis et al. (2002) used the same optimization scheme and reported average MSD values of 0.15 (Mg ha-') 2 and 0.24 (Mg ha-') 2 in North Carolina and Georgia, respectively. Colson et al. (1995) found higher MSD values, ranging from 0.79 to 1.28 (Mg ha-') 2 . Their study did not tune cultivar coefficients but used other direct field measurements. Our MSD values are slightly higher than those of 2.0-
Components of Mean Squared Deviation Maturity Group VI
-2.0
rLCS
15-
previous work that used direct solar radiation or soil moisture measurement or adjusted soil parameters or genetic coefficients to minimize observed vs. simulated yield. Of course, these detailed validation studies are required for model improvement. So too is evaluation of crop models across broad regions and over multiple years using inputs that are not directly measured because this is the context within which many climate impact studies use these important tools. The MSD is sensitive to large individual differences between observed and simulated yield, and our overall results reflect such sensitivity. In most of the years where we found large differences between observed and simulated yield, extreme weather events or other factors (e.g., lodging, pests, soil nutrient deficiencies, or disease) compromised model performance. CROPGRO-Soybean does not account for such factors that reduce observed yield. This leads to overprediction, which inflates the SB component of the MSD. Overestimation at Fairhope, where yield was reduced because of hurricane damage during several years, provides an obvious example. Extreme events also influence the LCS component by affecting the pattern of interannual variability. The example for maturity group VII at St. Joseph demonstrates that while mean yield and the magnitude of interannual variability of simulated and observed yield match well, poor model performance during only 1 or 2 yr leads to a higher LCS, inflating MSD (Fig. 7a). CROPGRO-Soybean performed well under a wide
1.
0
Jeanerette, LA, Maturity Group VI
5-
- 1.00s
-1.0
4-
-5
* simulated o observed
-4
-
CO
0.5-
-0.5
I
3-
0
8
-3
0
0
0
L77 o~~~~~~
0.0 ai
0
m
'
0.0
S
0
0
2-
0
0
-
-2
0 0o
-~~~~cl
1co
x
-1
0 0
Fig. 4. Components of mean squared deviation for maturity group VI. LCS, lack of correlation weighted by the standard deviations; SDSD, squared difference betwveen standard deviations; SB, squared bias.
-
I
I
I
I
I
I
co 0)
co 0
co 0)
cO 0)
co 0)
CD 0
I 0
I
I
I
I
0) 0)
C) 0)
0 0
0 0
-O
Fig.5. Time series of simulated and observed yield for maturity group VI at Jeanerette, LA.
AGRONOMY JOURNAL, VOL. 95, MAY-JUNE 2003
542
2 Table 5. Error statistics for maturity group VII and Ransom cultivar (SD, standard deviation; CV, coefficient of variation; r , coefficient of determination; d, index of agreement).
Maturity group VII
Fairhope
Crossville
Marion Junction
Blackville
Tifton
Alexandria
Jeanerette
St. Joseph
N Mean simulated yield, Mg ha-' Mean observed yield, Mg ha-' SD simulated yield, Mg ha-' SD observed yield, Mg ha-' CV simulated yield CV observed yield 2
23 3.07*** 2.42 0.90 0.72 0.29 0.30 0.77 0.70
16 2.08 1.86 0.92 0.45 0.44 0.24 0.49 0.72
21 1.72 1.53 0.90 0.68 0.52 0.44 0.17 0.55
11 1.48 1.28 0.80 0.29 0.54 0.23 0.31 0.65
8 3.27 2.84 1.35 1.11 OA1 0.39 0.89 0.79
10 2.43 2.17 0.90 0.80 0.37 0.37 0.70 0.69
10 2.37 2.20 0.85 0.84 0.36 0.38 0.74 0.69
9 1.88 1.83 1.15 1.07 0.61 0.58 0.71 0.82
,
d
*** Siguificantly different than observed (at = 0.001).
range of environmental conditions, despite the use of several inputs that were not measured directly in the field. The pattern of interannual variability, which was largely influenced by poor model simulation during individual years, contributed most significantly to MSDs between simulated and observed yield. In many of the individual years when the model overestimated yield, observed values were influenced by pests, lodging, soil nutrient deficiencies, or extreme weather events. Because most crop models currently do not consider these effects completely, it is unreasonable to expect that the models will perform well in every individual year. When such years are eliminated, MSD drops significantly. Model deviations also should be viewed within the context of specific applications. Changes in mean yield and the magnitude of yield variance often serve as inputs to agricultural economics models (Lambert et al., 1995; McCarl et al., 2000) used in long-term climate change studies. Some have argued that accurate prediction of the range of interannual yield variability is more important than estimating absolute yield values in an individ-
ual year because year-to-year variability has greater potential economic impact (Moen et al., 1994). While Haskett et al. (1995) found poor performance during individual years, they argued that an ability to successfully simulate soybean mean yield and long-term variability justifies model use for climate change studies. The results presented here suggest that CROPGROSoybean meets this criterion.
- *
0
LCS
0 0
()
o
-
-2 o
0
*-
o
-
-0 1-
0-
-1
I U)
I CD
a)
al
I 0Ca a) t-
co
I ca co a)
I N4 3) a)
I . 0) a)
I ° CD ax
] 0) co
I
I ° a) a)
C'
0) X)
-0
Fairhope, AL, Ransom
b
-5
-1.0
9) 1.0-
0* a')
00
S
5-
~0 CD
0
2-
XSB
0
-3
0 0
-0
-1.5
XSDSD
-4
3-
oo
.gu 1.5-
-5
simulated
4_ o observed
Components of Mean Squared Deviation -2 0 Maturity Group VIl
2.0-
St. Joseph, LA, Maturity Group VII
a
5-
0
4-
-
-
-0
-
~~~-0.5
VN 11
m 0.5-
co
00
0)
30
v.v -J
-J 0C
)
6 n J
