UNCLASSIFIED / APPROVED UNCLASSIFIED FOR PUBLIC RELEASE
A Design of Experiments approach to evaluating Parameterization Schemes for Numerical Weather Prediction. Jeffrey A. Smith, Ph.D. and Richard S. Penc, Ph.D. U.S. Army Research Laboratory RDRL-CIE-M WSMR, NM 88002-5501
[email protected] and
[email protected]
UNCLASSIFIED / APPROVED FOR PUBLIC RELEASE
The Nation’s Premier Laboratory for Land Forces
UNCLASSIFIED
Outline • • • • • • • •
Motivation Problem Model Parameterizations Design Space Complications Our Approach Summary and References
UNCLASSIFIED
The Nation’s Premier Laboratory for Land Forces
UNCLASSIFIED
Warfighter Motivation All Images Credit: E. Ebert
Mesoscale Forecast
Synoptic Forecast
Observed Rainfall
Q: Which rainfall forecast would a Warfighter rather use? A: The warfighter would likely prefer the “More Resolved” (left) forecast because it is more operationally relevant both in terms of intensity and spatial distribution. Q: Which is a “better” rainfall forecast? A: The forecaster would respond that the “Smooth” (center) forecast “wins” according to traditional model verification approaches. UNCLASSIFIED
The Nation’s Premier Laboratory for Land Forces
UNCLASSIFIED
The Modeler’s Dilemma Displacement Error by Grid Resolution
Image and data credit: Mass et al. 2002
Resolution (km)
UNCLASSIFIED
Mean Absolute Error (mb)
36
4.19
12
4.81
4
5.25
Modeling Tradeoffs
• The size of measurable features (e.g., thunderstorm) is driven by resolution; however, spatial and temporal resolution drives compute time and model error. • Finer resolutions produce more detail; however, sub-grid effects are poorly modeled at fine scales. • Model initialization and data assimilation allow for rapid model spin-up to local conditions; however, the Warfighter often fights in data sparse (poorly sensed from a meteorological standpoint) regions.
The Nation’s Premier Laboratory for Land Forces
UNCLASSIFIED
Our (Broad) Problem • •
•
• •
•
We do not know in advance where the Army will be deployed, we need a model that works well in many areas, vice one optimized for a single region. Many approaches (called parameterizations) exist to handle “sub-grid” effects, such as moisture, land use, etc. Under what conditions is one parameterization more useful than another? Current model evaluation studies randomly test combinations of parameterizations based on limited experience of others with the supplied physics packages Only a very small fraction of these combinations have been tested and reported in the peer-reviewed literature. We need a systematic, ordered approach to the evaluation process that will lead to the best possible combination for use in as many environments as possible. Under what conditions do other modeling parameters such as observation nudging weight or nesting ratio allow our model to perform robustly over a variety of locations and atmospheric conditions?
UNCLASSIFIED
The Nation’s Premier Laboratory for Land Forces
UNCLASSIFIED
Model (1/2) • There are 6 primitive equations that constitute a weather model. The following equation describes one, the conservation of mass: v where ρ is the density of air, (u, v, w) describe wind speed along three orthogonal axes, and (x, y, z) are the spatial coordinates along three orthogonal axes. • These non-linear partial differential equations predict future atmospheric states, generally out to ~ 6 hours, within a limited volume using a finite difference form of the equations of motion. ,
,
,
→
|
where represents the observational input up to time , , represents sources of randomness such as varied global (synoptic) conditions up to time , the collection of adjustable and parameters. Generally, up to time the model spins up, and after the model free runs to time t. UNCLASSIFIED
The Nation’s Premier Laboratory for Land Forces
UNCLASSIFIED
Model (2/2)
Inputs include: – Coarse grid interpolated model data that establish boundary conditions. This data comes from other global/large scale forecast models. – Supplemental observational data collected from deployed sensors augments the coarse grid input data through data assimilation techniques that nudge the model initialization toward “ground truth”. UNCLASSIFIED
The Nation’s Premier Laboratory for Land Forces
UNCLASSIFIED
Parameterizations • Microphysics – Moisture.
• Cumulus – Clouds.
• Radiation – Solar and terrestrial emissions.
• Planetary Boundary Layer (PBL) – Atmosphere land interface, turbulence, etc.
• Surface – Couples atmosphere to the land. Image attribution to J. Dudhia 2015
Interaction of parameterization schemes in the WRF-ARW weather modelling code along with select model variables and features. UNCLASSIFIED
• Diffusion (not shown) – Energy balance • Land Surface (not shown) – Models surface characteristics (urban, grasslands, etc.)
The Nation’s Premier Laboratory for Land Forces
UNCLASSIFIED
Design Space (Sampling
Parameterizations: – Factors with each level a different scheme (nominal) • • • •
Microphysics: 18 levels Cumulus : 11 levels Radiation: 9 levels Planetary Boundary Layer and Surface – 10 and 7 levels respectively; however, only 15 effective combinations.
• Diffusion • Land Surface Models: 7 levels
– On the order of 2M effective combinations for Parameterizations alone.
UNCLASSIFIED
using
)
Other “Parameters”: Region (inner domain center point) – Southern California, San Francisco Bay, (ordinal) …
Initialization Data Used – Radius and strength of influence (continuous) – Fraction of observations used to initialize (continuous) – Class of data uses (Surface, Upper Air, TAMDAR, … ) (nominal)
Nesting and Resolution – Inner grid resolution (0.5 km to 2km) (continuous) – Nest count (3 or 4) (ordinal) – Nesting ratio (3:1 up to 6:1) (ordinal)
The Nation’s Premier Laboratory for Land Forces
UNCLASSIFIED
Our Approach (developing
)
• Use Orthogonal Arrays (Hedayat et al., 1999) or Latin Hypercube (Kleijnen et al. 2005) designs – Create an orthogonal or nearly orthogonal design • • • •
Based on Known constraints on parameterization interactions Available computing resources Computing time
– To sample the model
at those design points.
• Recognize that we will likely add Levels to some factors (other domains for example) and possibly other Factors (assimilation methods) as time goes on. • Ideally, the first step is to screen the space of parameterizations to a small collection that gives robust performance. UNCLASSIFIED
The Nation’s Premier Laboratory for Land Forces
UNCLASSIFIED
Complications • Synoptic conditions (continental scale weather patterns such as the Jet Stream) can vary dramatically on the scale of days or months: – Speed of these changes influence domain consideration to allow ample time to downscale coarse inputs to fine-scale.
• Care must be taken in quality controlling the input meteorological data: – Data quality varies based on location of observing station, meso-net, proximity to collection time, etc.
• The time per run (~ 2h of wall time), the post-processing time, and the analysis time make exploring the entire space of combinations (~ 2M combinations of possible parameterizations alone) entirely prohibitive by any organization with finite resources. UNCLASSIFIED
The Nation’s Premier Laboratory for Land Forces
UNCLASSIFIED
Summary and Sources Summary
Selected Sources
• Numerical weather prediction codes are complex simulations of highly non-linear physical processes; yet are reasonably well modeled using coupled partial differential equations. • The general trend for these codes is towards ever smaller grid resolutions, a trend that can only benefit the warfighter if we can confirm the accuracy of these models. • Design of Experiments should allow us to better understand how these codes function, and thereby illuminate areas that need improvement; hence, sufficiently characterize the strengths of these models to the warfighter.
Bohlin T. 2006. Practical Grey-Box Process Identification: Theory and Applications. London: Springer-Verlag. Box GEP, Hunter WG, Hunter JS. 1978. Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building. New York: Wiley. Brown B, Jensen T, Gilleland E, Tollerud E, Ebert E. 2012. Spatial Verification Methods and Their Application to Ensemble Forecasts. 10th Meeting of the THORPEX GIFS-TIGGE Working Group. Boulder, CO. Dudhia J. 2015. Overview of WRF Physics. 2015 Basic WRF Tutorial. Boulder, CO: National Center for Atmospheric Research. Hedayat AS, Sloane NJA, Stufken J. 1999. Orthogonal Arrays: Theory and Applications New York: Springer. Kleijnen JPC, Sanchez SM, Lucas TW, Cioppa TM. 2005. A User's Guide to the Brave New World of Designing Simulation Experiments. INFORMS Journal on Computing 17(3):263-289. Mass CF, Ovens D, Westrick K, Colle BA. 2002. Does Increasing Horizontal Resolution Produce More Skillful Forecasts? Bulletin of the American Meteorological Society 83(3):407-430. Montgomery DC. 1997. Design and Analysis of Experiments. New York: Wiley. Pielke RA, Sr. 2002. Mesoscale Meteorological Modeling. San Diego: Academic Press. Skamarock WC, Klemp JB, Dudhia J, Gill DO, Barker DM, Duda MG, Huang X-Y, Wang W, Powers JG. 2008. A Description of the Advanced Research Wrf Version 3. Boulder CO. NCAR Technical Note No. NCAR/TN-475+STR. Stensrud DJ. 2007. Parameterization Schemes: Keys to Understanding Numerical Weather Prediction Models. Cambridge: Cambridge University Press. Warner TT. 2011. Numerical Weather and Climate Prediction. Cambridge: Cambridge University Press.
UNCLASSIFIED
The Nation’s Premier Laboratory for Land Forces
