A Design of Experiments approach to evaluating ... - Amazon AWS

UNCLASSIFIED / APPROVED UNCLASSIFIED FOR PUBLIC RELEASE

A Design of Experiments approach to evaluating Parameterization Schemes for Numerical Weather Prediction. Jeffrey A. Smith, Ph.D. and Richard S. Penc, Ph.D. U.S. Army Research Laboratory RDRL-CIE-M WSMR, NM 88002-5501 jeffrey.a.smith1.civ@mail.mil and richard.s.penc.ctr@mail.mil

UNCLASSIFIED / APPROVED FOR PUBLIC RELEASE

The Nation’s Premier Laboratory for Land Forces

UNCLASSIFIED

Outline • • • • • • • •

Motivation Problem Model Parameterizations Design Space Complications Our Approach Summary and References

UNCLASSIFIED


UNCLASSIFIED

Warfighter Motivation All Images Credit: E. Ebert

Mesoscale Forecast

Synoptic Forecast

Observed Rainfall

Q: Which rainfall forecast would a Warfighter rather use? A: The warfighter would likely prefer the “More Resolved” (left) forecast because it is more operationally relevant both in terms of intensity and spatial distribution. Q: Which is a “better” rainfall forecast? A: The forecaster would respond that the “Smooth” (center) forecast “wins” according to traditional model verification approaches. UNCLASSIFIED


UNCLASSIFIED

The Modeler’s Dilemma Displacement Error by Grid Resolution

Image and data credit: Mass et al. 2002

Resolution (km)

UNCLASSIFIED

Mean Absolute Error (mb)

36

4.19

12

4.81

4

5.25

Modeling Tradeoffs

• The size of measurable features (e.g., thunderstorm) is driven by resolution; however, spatial and temporal resolution drives compute time and model error. • Finer resolutions produce more detail; however, sub-grid effects are poorly modeled at fine scales. • Model initialization and data assimilation allow for rapid model spin-up to local conditions; however, the Warfighter often fights in data sparse (poorly sensed from a meteorological standpoint) regions.


UNCLASSIFIED

Our (Broad) Problem • •

•

• •

•

We do not know in advance where the Army will be deployed, we need a model that works well in many areas, vice one optimized for a single region. Many approaches (called parameterizations) exist to handle “sub-grid” effects, such as moisture, land use, etc. Under what conditions is one parameterization more useful than another? Current model evaluation studies randomly test combinations of parameterizations based on limited experience of others with the supplied physics packages Only a very small fraction of these combinations have been tested and reported in the peer-reviewed literature. We need a systematic, ordered approach to the evaluation process that will lead to the best possible combination for use in as many environments as possible. Under what conditions do other modeling parameters such as observation nudging weight or nesting ratio allow our model to perform robustly over a variety of locations and atmospheric conditions?

UNCLASSIFIED


UNCLASSIFIED

Model (1/2) • There are 6 primitive equations that constitute a weather model. The following equation describes one, the conservation of mass: v where ρ is the density of air, (u, v, w) describe wind speed along three orthogonal axes, and (x, y, z) are the spatial coordinates along three orthogonal axes. • These non-linear partial differential equations predict future atmospheric states, generally out to ~ 6 hours, within a limited volume using a finite difference form of the equations of motion. ,

,

,

→

|

where represents the observational input up to time , , represents sources of randomness such as varied global (synoptic) conditions up to time , the collection of adjustable and parameters. Generally, up to time the model spins up, and after the model free runs to time t. UNCLASSIFIED


UNCLASSIFIED

Model (2/2)

Inputs include: – Coarse grid interpolated model data that establish boundary conditions. This data comes from other global/large scale forecast models. – Supplemental observational data collected from deployed sensors augments the coarse grid input data through data assimilation techniques that nudge the model initialization toward “ground truth”. UNCLASSIFIED


UNCLASSIFIED

Parameterizations • Microphysics – Moisture.

• Cumulus – Clouds.

• Radiation – Solar and terrestrial emissions.

• Planetary Boundary Layer (PBL) – Atmosphere land interface, turbulence, etc.

• Surface – Couples atmosphere to the land. Image attribution to J. Dudhia 2015

Interaction of parameterization schemes in the WRF-ARW weather modelling code along with select model variables and features. UNCLASSIFIED

• Diffusion (not shown) – Energy balance • Land Surface (not shown) – Models surface characteristics (urban, grasslands, etc.)


UNCLASSIFIED

Design Space (Sampling

Parameterizations: – Factors with each level a different scheme (nominal) • • • •

Microphysics: 18 levels Cumulus : 11 levels Radiation: 9 levels Planetary Boundary Layer and Surface – 10 and 7 levels respectively; however, only 15 effective combinations.

• Diffusion • Land Surface Models: 7 levels

– On the order of 2M effective combinations for Parameterizations alone.

UNCLASSIFIED

using

)

Other “Parameters”: Region (inner domain center point) – Southern California, San Francisco Bay, (ordinal) …

Initialization Data Used – Radius and strength of influence (continuous) – Fraction of observations used to initialize (continuous) – Class of data uses (Surface, Upper Air, TAMDAR, … ) (nominal)

Nesting and Resolution – Inner grid resolution (0.5 km to 2km) (continuous) – Nest count (3 or 4) (ordinal) – Nesting ratio (3:1 up to 6:1) (ordinal)


UNCLASSIFIED

Our Approach (developing

)

• Use Orthogonal Arrays (Hedayat et al., 1999) or Latin Hypercube (Kleijnen et al. 2005) designs – Create an orthogonal or nearly orthogonal design • • • •

Based on Known constraints on parameterization interactions Available computing resources Computing time

– To sample the model

at those design points.

• Recognize that we will likely add Levels to some factors (other domains for example) and possibly other Factors (assimilation methods) as time goes on. • Ideally, the first step is to screen the space of parameterizations to a small collection that gives robust performance. UNCLASSIFIED


UNCLASSIFIED

Complications • Synoptic conditions (continental scale weather patterns such as the Jet Stream) can vary dramatically on the scale of days or months: – Speed of these changes influence domain consideration to allow ample time to downscale coarse inputs to fine-scale.

• Care must be taken in quality controlling the input meteorological data: – Data quality varies based on location of observing station, meso-net, proximity to collection time, etc.

• The time per run (~ 2h of wall time), the post-processing time, and the analysis time make exploring the entire space of combinations (~ 2M combinations of possible parameterizations alone) entirely prohibitive by any organization with finite resources. UNCLASSIFIED


UNCLASSIFIED

Summary and Sources Summary

Selected Sources

• Numerical weather prediction codes are complex simulations of highly non-linear physical processes; yet are reasonably well modeled using coupled partial differential equations. • The general trend for these codes is towards ever smaller grid resolutions, a trend that can only benefit the warfighter if we can confirm the accuracy of these models. • Design of Experiments should allow us to better understand how these codes function, and thereby illuminate areas that need improvement; hence, sufficiently characterize the strengths of these models to the warfighter.

Bohlin T. 2006. Practical Grey-Box Process Identification: Theory and Applications. London: Springer-Verlag. Box GEP, Hunter WG, Hunter JS. 1978. Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building. New York: Wiley. Brown B, Jensen T, Gilleland E, Tollerud E, Ebert E. 2012. Spatial Verification Methods and Their Application to Ensemble Forecasts. 10th Meeting of the THORPEX GIFS-TIGGE Working Group. Boulder, CO. Dudhia J. 2015. Overview of WRF Physics. 2015 Basic WRF Tutorial. Boulder, CO: National Center for Atmospheric Research. Hedayat AS, Sloane NJA, Stufken J. 1999. Orthogonal Arrays: Theory and Applications New York: Springer. Kleijnen JPC, Sanchez SM, Lucas TW, Cioppa TM. 2005. A User's Guide to the Brave New World of Designing Simulation Experiments. INFORMS Journal on Computing 17(3):263-289. Mass CF, Ovens D, Westrick K, Colle BA. 2002. Does Increasing Horizontal Resolution Produce More Skillful Forecasts? Bulletin of the American Meteorological Society 83(3):407-430. Montgomery DC. 1997. Design and Analysis of Experiments. New York: Wiley. Pielke RA, Sr. 2002. Mesoscale Meteorological Modeling. San Diego: Academic Press. Skamarock WC, Klemp JB, Dudhia J, Gill DO, Barker DM, Duda MG, Huang X-Y, Wang W, Powers JG. 2008. A Description of the Advanced Research Wrf Version 3. Boulder CO. NCAR Technical Note No. NCAR/TN-475+STR. Stensrud DJ. 2007. Parameterization Schemes: Keys to Understanding Numerical Weather Prediction Models. Cambridge: Cambridge University Press. Warner TT. 2011. Numerical Weather and Climate Prediction. Cambridge: Cambridge University Press.

UNCLASSIFIED
