AMIP Simulation with the CAM4 Spectral Element Dynamical ... - CGD

Generated using V3.0 of the official AMS LATEX template–journal page layout FOR AUTHOR USE ONLY, NOT FOR SUBMISSION!

AMIP Simulation with the CAM4 Spectral Element Dynamical Core K.J. Evans

P. H. Lauritzen, S. Mishra, R. B. Neale

Oak Ridge National Laboratory, Oak Ridge TN

National Center for Atmospheric Research, Boulder, CO

M.A. Taylor∗

J.J. Tribbia

Sandia National Laboratories, Albuquerque, NM

National Center for Atmospheric Research, Boulder, CO

ABSTRACT We evaluate the climate produced by the Community Climate System Model, version 4, running with the new spectral-element atmospheric dynamical core option. The spectral-element method is configured to use a cubed-sphere grid, providing quasi-uniform resolution over the sphere, increased parallel scalability and removing the need for polar filters. It uses a fourth order accurate spatial discretization which locally conserves mass and total energy. Using the Atmosphere Model Intercomparison Project protocol, we compare the results from the spectral-element dynamical core with those produced by the default finite-volume dynamical core and with observations. Even though the two dynamical cores are quite different, their simulated climates are remarkably similar. When compared to observations, both models have strengths and weaknesses, but have nearly identical root-mean-square errors and the largest biases show little sensitivity to the dynamical core. The spectral element core does an excellent job reproducing the atmospheric kinetic energy spectra, including fully capturing the observed Nastrom-Gage transition when running at 0.125◦ resolution.

1. Introduction

2011) which allows different grids for the atmosphere, land and ice/ocean components. The spectral element method was chosen for integration into CAM based on its demonstrated scalability when run as a standalone dynamical core. The high-order finite element structure of the method also makes it well suited to upcoming heterogeneous computer architectures (Carpenter et al. 2011). In addition to scalability, the spectral element method is known to produce accurate solutions for atmospheric problems, as demonstrated first with shallow water test cases (Taylor et al. 1997; Thomas and Loft 2002), three-dimensional dry dynamical test cases (Taylor et al. 1998; Thomas and Loft 2005; Dennis et al. 2005; Taylor et al. 2007; Lauritzen et al. 2010), multicloud simulations (Khouider et al. 2010), aqua planet experiments which include full physics (Taylor et al. 2008; Mishra et al. 2011b,a) and realistic simulations with CAM2 physics (Wang et al. 2007). The SE method has also been pursued for global forecast modeling, such as Giraldo and Rosmond (2004); Giraldo (2005); Kim et al. (2008). The goal of this work is to build confidence in CAM-SE to justify devoting future model development and computational resources targeting high resolution simulations that are made possible by the parallel scalability of CAM-SE. We first give an overview of the spectral element dynamical core used in CAM-SE (Sec. 2), and then compare CAM4 simulations produced by CAM-SE, CAM-FV, and observations. We compare the kinetic energy spectra (Sec. 5),

Here we report on the climate produced by a spectralelement atmospheric dynamical core option within the Community Climate System Model (CCSM) version 4. The CCSM is a state of the art climate model with atmosphere, ocean, land and ice component models that exchange information through a flux coupler (Gent et al. 2011). The spectral-element dynamical core comes from HOMME, the High-Order Method Modeling Environment (Dennis et al. 2005, 2011), which has been integrated into the CCSM Community Atmosphere Model (CAM). The standalone HOMME model is also used for research into other numerical methods, such as discontinuous-Galerkin (Nair 2009), adaptive mesh refinement (St.-Cyr et al. 2008) and implicit time integration (Evans et al. 2010). We refer to CAM with the spectral-element dynamical core as CAMSE, CAM with the default finite volume dynamical core (Neale et al. 2011) as CAM-FV, and CAM with the Eulerian global spectral method (Collins, W. D. et al. 2006) option as CAM-EUL. CAM-SE is capable of using fully unstructured grids. Support for unstructured grids and the integration of the spectral element method in CAM was made possible by the decoupling of dynamics and physics used in CAM’s process split approach (Williamson 2002), the extensive infrastructure work to support unstructured grids (Worley and Drake 2005), and the CCSM tri-grid capability (Craig et al. 1

CESM1 F1850, ATM component, BGP

Simulated Years/Day

8

4 SE 0.25° FV 0.25° EUL T341

2

1

0.5

0.25

1K

4K

16K NCORES

64K

256K

Fig. 2. Performance of the CCSM atmosphere component model on Intrepid (IBM BG/P) when using the CAM-SE, FV or EUL dynamical core, showing the simulated-yearsper-day as a function of the number of processing cores. Atmosphere component times taken from a CCSM timeslice simulation, coupling the atmosphere (at 0.25◦ or T341 resolution), the land model (0.25◦ resolution), and the sea ice and data ocean model (0.1◦ ). The solid black line shows perfect parallel scalability. When using CAM-SE, the CCSM achieves near perfect scalability down to one element per processor, running at 12.2 SYPD on 86,400 cores.

Fig. 1. An example quadrilateral element grid for the sphere. An inscribed cube is projected on the surface of the sphere. The faces of the cubed-sphere are further subdivided to form a quadrilateral grid of the desired resolution. Coordinate lines from the Gnomonic equal-angle projection are shown. mean climate (Sec. 6) and discuss some transient features (Sec. 7) and regional climate details (Sec. 8), using 1◦ or equivalent resolutions. For the spectra, we also include results from aqua planet and the CAM-EUL simulations at up to 0.125◦ resolutions. 2. The CAM Spectral Element Dynamical Core CAM-SE uses a continuous Galerkin spectral finite element method (Taylor et al. 1997; Fournier et al. 2004; Thomas and Loft 2005; Wang et al. 2007; Taylor and Fournier 2010), commonly referred to as the spectral element method. These class of methods were designed from inception for fully unstructured grids. For the simulations presented here, the goal is global uniform resolution so we use the cubed-sphere grid (Fig. 1) first used in Sadourny (1972), with the equal angle projection (Ranˇci´c et al. 1996). This represents a large change in the horizontal grid as compared to CAM’s other dynamical core options, but most other aspects of CAM-SE are based on a combination of welltested approaches from CAM-EUL and CAM-FV. CAMSE tracer transport is modeled closely on CAM-FV; it uses the same conservation form of the transport equation and the same vertically Lagrangian discretization (Lin 2004). CAM-SE dynamics (momentum, temperature and surface pressure) are modeled closely on CAM-EUL; they share the same vertical coordinate, vertical discretization, hyperviscosity based horizontal diffusion, top-of-model dissipation, and solve the same moist hydrostatic equations. CAM-

2

SE (like FV) uses the vector-invariant form of the momentum equation instead of the vorticity-divergence formulation. The CAM-SE discretization provides several benefits compared to CAM-FV and CAM-EUL. As with all methods on quasi-uniform grids, avoiding the inherently load-imbalanced polar filter allows for efficient, fixed two-dimensional domain decomposition, resulting in significantly improved scaling on parallel computers. Such improved scaling was one of the motivations for migration to the cubed-sphere grid in Donner et al. (2011), and improved scaling in CAM is also expected from FV methods designed for unstructured grids (Putman and Lin 2007, 2009; Ringler et al. 2011). CAM-SE also improves CAM’s numerical conservation: it locally conserves (at the element level) mass, tracer mass and moist total energy without the use of ad-hoc fixers, and the horizontal momentum advection operator locally conserves potential vorticity (PV). Mass and PV conservation is to machine precision, while energy conservation is to the level of time-truncation error and non-adiabatic processes in the advection of specific humidity. The energy dissipation introduced by these terms is less than 0.02 W/m2 (Taylor 2011). We have compared the computational performance of CAM-SE, FV and EUL at high-resolution (0.25◦ and 0.125◦ )

using the CCSM running in prescribed SST mode in Dennis et al. (2011). We summarize the 0.25◦ results in Fig. 2. The figure shows the atmosphere component times only so that we can directly compare the different dynamical core options. It ignores the cost of the other components, which contribute from 25–50% of the total cost of the CCSM. CAM-EUL remains the most efficient model at low processor counts, achieving 0.8 simulated-years-per-day (SYPD) on only 2048 cores. But as the number of cores is increased, the performance of CAM-EUL plateaus at 0.87 SYPD. CAM-SE and CAM-FV are similar in performance low processor counts, but CAM-SE has near perfect scalability to 86,400 cores, achieving a top speed of 12.2 SYPD, while CAM-FV is less scalable and achieves its best performance of 2.49 SYPD on 53,248 cores. CAM-EUL uses a parallel strategy which makes use of multiple domain decompositions depending on the resolution and number of processors (Worley and Drake 2005). The column physics computations rely on a horizontal domain decomposition where each processor contains the entire vertical column of data for each horizontal grid point. The FFTs (in longitude) use a decomposition where each processor contains complete latitude bands, and the Legendre transforms (in latitude) uses a decomposition where each processor contains complete longitude bands. At high processor counts, the communication to transpose between these different data decompositions eventually becomes the dominant cost of the model, causing the lack of scalability seen in Fig. 2. CAM-FV improves on the scalability of CAM-EUL because it does not require a data decomposition to support Legendre transforms, but it still requires decompositions to support the physics and the polar filter. CAM-SE uses a single decomposition across elements for both physics and dynamics. It thus requires no data transpose operations to move data between multiple decompositions and instead relies solely on nearest neighbor communication. The parallel performance of CAM-SE also benefits from the spectral element Galerkin formulation (Dennis et al. 2011). The dynamics timestep is implemented as a twostep process. Step 1 contains all numerical calculations and is completely local to each element. Step 2 is the application of the spectral element projection operator (equivalent to the traditional finite element inverse-mass matrix). These two steps give a natural separation between computation and parallel communication. The step 1 computations are performed one element at a time and only require information stored at the Gauss-Lobatto nodes within each element. No inter-processor communication, ghost-cell or other neighbor information is needed. All communication is isolated to step 2, in which only element boundary data must be exchanged between adjacent elements. As all communication is isolated to a single projection operator, one needs only focus on load balancing strategies and message

Fig. 3. A 4×4 tensor product grid of Gauss-Lobatto nodes used within each element, for a degree d = 3 discretization. Nodes on the boundary are shared by neighboring elements. scheduling designed for this single operator. Significant resources have been devoted to optimizing this operator (Loft et al. 2001; Dennis 2003). Performance benchmarks for an ultra-high resolution fully coupled CCSM, running a 0.125◦ atmosphere and 0.1◦ ocean are given in Worley et al. (2011), where the CCSM obtains 2.6 SYPD using 204,408 cores. In that run, CAMSE was the dominant component, using 172,800 core running at 4.7 SYPD. For the moderate resolution (1◦ ) AMIP configuration described below, we have made simulations on the Cray XT5 system at Oak Ridge National Laboratory. Using 912 cores (76 nodes), the CCSM obtains 16.5 SYPD, and the CAM-SE component represents 61% of the total cost, running at 27.2 SYPD. Within the atmosphere, the dynamics represents 72% of the total cost. 3. CCSM AMIP Configuration Most of the simulations presented here use the Atmosphere Model Intercomparison Program (AMIP) protocol (Gleckler 2004), running at ∼ 1◦ resolution. We use the CCSM Version 4, coupling CAM with the 1◦ FV Community Land Model (CLM), the Community Ice CodE (CICE) with prescribed ice extent, version 4, and a data ocean model that provides prescribed sea surface temperature (SST) data through the same coupler infrastructure as would be used by an active ocean model. The SST data is from the 1◦ Hadley dataset for time period 1979 to 2000 (Rayner et al. 2005). The CAM-FV simulations use a 0.9 × 1.25 latitude-longitude grid and are further described in Neale et al. (2011). CAM-SE is using a cubedsphere grid where each cube face has 30×30 elements. The variables within each element are represented by polynomials up to degree 3, sampled on a 4 × 4 tensor product Gauss-Lobatto quadrature grid (Fig. 3). Counting all the non-overlapping quadrature points of the functions within each element across all elements, the CAM-SE grid has an average grid spacing at the equator of 1◦ and a minimum grid spacing of 0.83◦ . For this study, the CAM-SE AMIP simulation was configured to be as close as possible to CAM-FV. The main difference is that the CAM-SE simulations are performed

3

with a tri-grid configuration (Craig et al. 2011). In a trigrid configuration, the CCSM coupler is used to couple the atmosphere (running on the cubed-sphere) with the CLM (running on a 0.9 × 1.25 latitude-longitude grid, also used by CAM-FV) and the data ocean model (running on a 1◦ CCSM ocean grid). between these grids, the coupler uses conservative mapping weights generated by the SCRIP software package (Jones 1999). Thus, both CAM simulations use identical CLM configuration and SST datasets, but due to interpolation there will be slight differences in how the fluxes are represented on the atmosphere grid. All the simulations use the identical CAM Version 4 physics (Neale et al. 2010, 2011), with a 30 minute physics timestep and 26 vertical levels with a model top at 2.2hPa. CAM-SE uses the default CAM-EUL T85 tunings, with the exception of low and high cloud relative humidity threshold parameters (rhminl and rhminh respectively). These parameters have been determined so as to produce an energy balanced simulation at the top of the atmosphere when running with 1850-like conditions. The globally averaged annual energy budgets are sensitive to these values (Williamson et al. 1995; Stratton 1999; Jackson et al. 2008), because they represent the minimum threshold for the formation of low and high clouds over a resolution dependent grid box area, and they must be chosen carefully. The values rhminl=0.91 and rhminh=0.78 were selected to produce a global balance of shortwave and longwave energy entering and exiting the atmosphere at the model top and surface. Also, they were set such that the zonally averaged level of shortwave and longwave energy flux through the vertical model boundaries was very close to CAM-FV predindustrial simulations using default physics parameter settings relative to observations (not shown). For reference, the CAM-FV at 1◦ resolution uses rhminl=0.92 and rhminh=0.77. With respect to the remaining tunable parameter, the only differences between CAM-SE and CAM-FV (due to the fact that the CAM-SE tunings were taken from the CAM-EUL default values) are in the cldwat icritc, cldwat icritw, zmconv c0 lnd and zmconv c0 ocn parameters described in Neale et al. (2010). The values for CAM-SE were 16 × 10−6 , 4 × 10−4 , 0.004 and 0.004, while CAM-FV uses 18 × 10−6 , 2 × 10−4 , 0.0035 and 0.0035, respectively. 4.

In Lauritzen et al. (2010), baroclinic instability triggered by grid anisotropy is compared across 8 different dynamical cores, including cores running on cubed-sphere and icosahedral grids. Of the non-latitude/longitude grids, CAMSE shows the least amount of grid imprinting. A more stringent test looks at the climate in aqua planet simulations with full physics. When run with zonally symmetric boundary conditions and forcing, any grid anisotropy would be expected to show up in long time averages where all transient flow features have been averaged out. In Taylor et al. (2008), the time averaged vertical pressure velocity (one of the noisiest fields) is shown to have no visible grid imprinting. It does show some departures from zonal symmetry, but these are at the same level as in the (perfectly isotropic) CAM-EUL model. We attribute CAM-SE’s low levels of grid imprinting to the use of relatively high order numerics and the fact that finite element methods have a long history of dealing with grids far more unstructured and anisotropic than the cubed-sphere. With respect to the quadrature nodes, we note that their use is similar to the quadrature grid used in CAM-EUL: they are used to evaluate the integrals that appear in Galerkin methods, but the underlying representation (polynomials in the case of CAM-SE and spherical harmonics in the case of CAM-EUL) does not contain the non-uniformities seen in the quadrature points. 5. Kinetic Energy Spectra A comparison of the horizontal kinetic energy spectra among the CAM dynamical cores is one way to quantify solution accuracy and characterize the dissipation properties of the dynamical core (Skamarock 2004, 2011). We compute the spectra from global 250hPa velocity ~u snapshots taken every 6 days, via a vector spherical harmonic transform (Adams and Swarztrauber 1997). For CAM-SE, the data is first interpolated from the cubed-sphere grid to a latitude/longitude grid using the element shape functions within each element. For vectors, we interpolate their components expressed in the Gnomonic contra-variant coordinate system and then map the result to spherical coordinates. This avoids the interpolation issues that arise when trying to interpolate individual latitude and longitude components which are discontinuous at the poles. Given a vector spherical harmonic expansion in harmonics of degree k and mode m, we consider the usual spectra E(k) giving the energy in the harmonics of degree k, summed over m. At large scales, the atmosphere behaves as a quasi two-dimensional flow with a strong enstrophy cascade, resulting in a −3 scaling, E(k) ∼ k −3 , matching observations (Nastrom and Gage 1985; Lindborg 1999). At smaller scales, below several hundred kilometers, the atmosphere has a robust transitions into a −5/3 regime which is less well understood. For global models, this Nastrom-

Grid Imprinting

Grid imprinting is of special concern for methods on non-latitude/longitude grids, as any such grid will have some anisotropy and must contain some special points. For the cubed-sphere grid, the main concern is anomalous forcing of an m = 4 mode caused by the 8 special points at the cube corners. In addition the use of Gauss-Lobatto nodes within each element (Fig. 3) are non-equally spaced. For CAM-SE, this issue has been studied extensively.

4

Fig. 4. The kinetic energy spectra from CAM-SE, CAMFV and CAM-EUL at ∼ 1◦ resolution.

Fig. 6. The kinetic energy spectra from high resolution aqua planet simulations of CAM-SE and CAM-EUL. rately. All three models approach the −3 scaling. The differences between the models are more easily seen in Fig. 5, which plots k 3 E(k) so that the −3 scaling is represented by a horizontal line. One can see that CAM-EUL is by far the least dissipative model, capturing the -3 scaling for almost all resolved scales. CAM-SE is closer in character to CAM-EUL than to CAM-FV, showing good agreement with CAM-EUL for most wave numbers and only departing from CAM-EUL as the resolution approaches the truncation limit. The results for CAM-FV and CAM-EUL are consistent with that seen in idealized aqua planet simulations (Jablonowski and Williamson 2011). CAM-FV departs from the −3 scaling at relatively low wave numbers. With respect to the divergent component, CAM-SE and CAM-EUL are very similar, while the CAM-FV spectra shows the impact of its use of a 2’nd order divergence damping (Lauritzen et al. 2011) starting around wave number 15. We expect that with the more scale selective 4’th order divergence damping used in the cubed-sphere FV core (Donner et al. 2011), CAM-FV would behave more like CAM-EUL and CAM-SE, as the later two models both rely on a 4’th order hyperviscosity operator.

Fig. 5. As in Fig. 4, except plotting k 3 E(k) to better illustrate how the spectra matches the predicted k −3 scaling (black line). Solid lines show the KE of ~u, while dotted lines show the divergent component ~uι . Gage transition is only captured at very high resolutions, requiring O(10km) grid spacing and a low dissipation numerical method (Hamilton et al. 2008).

b. a.

Moderate Resolution Spectra

High Resolution Spectra

We also show the spectra from high resolution (0.25◦ and 0.125◦ ) aqua planet simulations (Neale and Hoskins 2000a,b) in Figures 6 and 7, averaged over 1 year of data (72 snapshots, taken every 5 days) using the CAM as configured in Williamson (2008a,b). Averaging over 72 snapshots has eliminated much of the noise that was present in the low-resolution AMIP spectra in Fig. 4. Although this data is from aqua planet simulations, preliminary results indicate that the spectra from high-resolution AMIP simulations will be quite similar. In the low resolution results, the CAM-SE spectra rolled off earlier than the CAM-EUL

In Figures 4 and 5 we show the spectra from ∼ 1◦ resolution AMIP simulations. The spectra is computed from 1 month of instantaneous flow snapshots (5 snapshots, every 6 days) and then averaged. We plot the spectra for ~u (solid lines) and the divergent (irrotational) component ~uι (dotted lines). The divergent component is defined via the the Helmholtz decomposition, ~u = ~uι + ~us with ∇ × ~uι = 0 and ∇ · ~us = 0. At these scales, almost all the energy is contained in the rotational (solenoidal) ~us component, but it is illustrative to look at the divergent component sepa-

5

Table 1. Global Annually Averaged values of key atmospheric variables that govern energy balance for 1◦ CAMSE and CAM-FV simulations, along with observed quantities. Variable RESTOM FSNT(TOA) CLDTOT CLDHGH CLDLOW SWCF LWCF TS U 200hPa

Fig. 7. As in Fig. 6, only plotting k 5/3 E(k) to better illustrate how the spectra matches the predicted k −3 and k −5/3 scalings (black lines). Solid lines show the KE of ~u, while the dotted lines show the divergent component ~uι . Only the highest resolution, CAM-SE at 0.125◦ , has regions matching both scaling regime.

CAM-SE 0.84 243.3 48.8 23.6 35.1 -47.5 24.3 288.1 16.3

CAM-FV 1.03 242.2 47.0 23.7 31.5 -48.2 25.7 287.9 15.7

OBS 1.1 240.4 66.8 21.3 26.4 -47.1 29.9 287.7 15.3

CAM-SE and 0.937 for CAM-FV, showing CAM version 4 represents an 8% improvement in RMSE over version 3.5, and CAM-SE obtains a slight improvement over CAM-FV. Similar results are obtained for the bias, where CAM-SE and CAM-FV have biases of 0.839 and 0.905, respectively. We present global annually averaged values of some key variables and corresponding observed values in Table 1. The variables RESTOM, FSNT(TOA), CLDTOT, CLDHGH, CLDLOW, SWCF, and LWCF refer to the residual energy flux at the model top, the net absorbed shortwave energy at the top of the atmosphere, the fractional coverage of total, high, and low clouds, and the short and longwave cloud forcing respectively (units of W/m2 ). The variables TS and U200hPa refer to the global annually averaged surface temperature and the annually averaged 200hPa zonal wind field for the duration of the AMIP run. Observational values used for comparison come from the following datasets: CERES (Loeb et al. 2009) for FNST(TOA), LWCF, and SWCF, NCEP reanalysis (Kalnay et al. 1996; Kistler et al. 2001) for the surface temperature and 200hPa zonal wind, and ISCCP D2 (Rossow and Schiffer 1999) for the cloud (CLDTOT, CLDHGH, and CLDLOW) fields. Overall, the simulations produce a similar global climate, relative to observations. The RESTOM for both models are within the 1.1 ± 0.4W m−2 estimate from a collection of observations of the Earth’s energy balance at the top of the atmosphere from 1970 to 2000 (Murphy et al. 2009). For five of the nine variables listed in Table 1, the two models produce global annually averaged values closer to each other than observations. Note that due to overlap and a separate accounting of mid-level clouds, the high and low clouds to not sum up to CLDTOT. As presented in Table 1, the dynamically-based prognostic variables in CAM-SE such as temperature and wind speed match very closely to CAM-FV as well as observations. Fig. 8 shows the annually averaged sea-level pres-

spectra, but in the high resolution case, the figure shows that the CAM-SE 0.25◦ and CAM-EUL T341 simulations are quite similar, and agree well up to the maximum wave number resolved by the T341 simulation. We note that the T341 simulation uses a grid of size 512 × 1024 with an Equatorial grid spacing of 0.35◦ . At this resolution there is some hint of the Nastrom-Gage -3 to -5/3 transition, but it is not fully resolved until the resolution is increased to 0.125◦ . The CAM-SE 0.125◦ simulation has both a 3 regime as well as fully resolved -5/3 regime, best seen in Fig. 7. We expect that CAM-EUL at T720 would obtain a similar result. However, in the -5/3 regime, one can see that the magnitude of the divergent component is now significant (in Fig. 7, with a value of 30 out of a total of 80). Resolving this component of the energy is a requirement for capturing the Nastrom-Gage transition, suggesting that the 2’nd order divergence damping would not be appropriate at these resolutions. 6. Mean Climate We now turn to the mean climate (1981-2000) of CAMSE and CAM-FV 1◦ resolution AMIP simulations. The data presented in this section is computed from the model output by the publicly released CAM Atmosphere Model Working Group diagnostics package. The package computes an overall error measure from the Taylor diagram (Taylor 2001), from ten key diagnostics (including sea level pressure, cloud forcing, rainfall, temperature, wind stress, zonal wind and relative humidity) and normalized by the results from the previous CAM, version 3.5. The root mean square error (RMSE), not including the bias, is 0.920 for

6

Fig. 9. Annual implied northward ocean heat transport (in petawatts) of the total ocean for the CAM-SE and FV AMIP runs compared to the NCEP derived data (Trenberth and Caron 2001). the most complete precipitation dataset particularly over the oceans. It is a merged product of satellite, gauge and sounding data and is the most relevant dataset choice for lower resolution global climate model studies. Fig. 10 shows that the models are much closer to each other than to the observations. The RMSE between the two models is 0.55, while their RMSE with respect to observations is 1.15 and 1.13 for CAM-SE and CAM-FV respectively. They both significantly overestimate the average maximum precipitation rates in the tropics (a more significant analysis is given in section 7 below). The difference plots in Fig. 11 show that regionally there are significant biases in both models over most of the Pacific ITCZ, the west and east coasts of India, and over the tropical warm pool over the West Pacific. There are several localized biases in the CAM-SE simulation over the tropical Pacific warm pool, specifically near Jakarta, Indonesia, Papua New Guinea, and Western Colombia. There are also several regions where the biases are lower with SE than FV, namely off the West and East coasts of India. These relative model biases are within regions of strong overall precipitation biases that reach ∼ 6 mm/day. The model-produced spurious precipitation features near the Tibetan plateau (visable in Fig. 11) are also common to past versions of CAM. Alternative precipitation data sets including the CPC Merged Analysis of Precipitation CMAP (a merged satellite only product) and Tropical Rainfall Measuring Mission TRMM-3B42 (a merged satellite whose native resolution is 25 km) show similar observed features in this region. So this precipitation bias remains a distinct CAM problem related to insufficient blocking effects from steep orography when using such low horizontal

Fig. 8. Annually averaged sea level pressure for CAM-SE, CAM-FV and NCEP reanalysis. sure (SLP) values for CAM-SE and CAM-FV covering the AMIP period, and the overall distribution including the robust minima over the North Pacific and North Atlantic is captured well. The North Pacific low is more robust with CAM-SE than CAM-FV and observations, but its overall Arctic pressure minima is slightly closer to observations. The global horizontal distribution of annually averaged SLP in CAM-SE has a mean value of 1011.12, compared to 1011.25 for FV and 1011.62 from NCEP reanalysis. One robust example of the similarity of CAM-SE and CAM-FV is the total annual implied northward ocean heat transport. Fig. 9 shows that as observed, most of the transport occurs in the tropical regions. Both models produce transport values very close to each other and at levels near the lower edge of the error of the estimate (Trenberth and Caron 2001). The statistics of the hydrological cycle within CAM-SE and CAM-FV are quite similar. The precipitation rate is 3.0 mm/day with CAM-SE and 2.9 mm/day with CAMFV, compared to a global annual average of 2.6 mm/day from Global Precipitation Climatology Project (GPCP) data (Adler and Coauthors 2003). GPCP precipitation is 7

Fig. 11. Precipitation rate difference plots between CAMSE and CAM-FV, and between CAM-SE vs. GPCP. reach maximum amplitude, are displayed in Fig. 14 for both models and NCEP observations. Unlike Figures 12 and 13, which show the seasonal averages, Fig. 14 shows the annually averaged values and thus highlights more general behavior of the winds. The RMSE between the two models for this field is 1.32, which is less than half the RMSE of the models compared to NCEP observations, which are 2.87 and 2.52 for CAM-SE and CAM-FV respectively. The difference plot (not shown) indicates the higher differences of CAM-SE is due to the overestimation of the Southern Hemisphere tropospheric jet. This is consistent with the surface wind stress in the same location, as in Fig. 16. CAM-FV is slightly closer to observations at 200hPa, although the differences between the models are small relative to the uncertainty of the observations. Both models are also a significant improvement over version 3 of CAM, which was not able to capture the structure of the observed jets with as much fidelity (Hurrell et al. 2006). Other improvements in CAM from version 3 to 4 are documented in Gent et al. (2011). Fig. 15 shows the annual zonally averaged vertical pressure velocities for CAM-SE, CAM-FV, and NCEP observations. This field is sensitive to the smoothness of the topography datasets. The surface topography is necessarily represented as an approximation to the observed topography, and the roughness of this field impacts the diagnosed vertical pressure velocity. At present, CAM-SE is using a smoother topography dataset in order to reduce the level of noise in the vertical velocity, as can be seen in the figure. To compare the amount of orographic smoothing across dynamical cores, we look at the power spectra E(k)

Fig. 10. Precipitation rate from CAM-SE, CAM-FV and GPCP. resolutions. Addressing these problems at these resolutions is an active area of research and is expected to involve more accurately representing blocked flows and form drag due to unresolved sub-grid scale orogaphic features. The annually averaged zonally averaged zonal wind fields with CAM-SE show close agreement to CAM-FV compared to NCEP observations. We display the seasonally averaged zonal wind fields (Fig. 12 and 13) because the polar winter jet stream governs many important aspects of global climate, including the meridional transport of heat, the frequency and location of storm tracks, and the global distribution of surface pressure and temperature. Fig. 12 shows the zonally averaged zonal wind field averaged over December-February (DJF) for CAM-SE, CAMFV and NCEP observed values covering the AMIP period. Both CAM-SE and CAM-FV overestimate the easterly flow near the equator, however the CAM-SE is able to represent the stratospheric Arctic winter jet better than FV, which is too strong by ∼ 9 m/s. The stratospheric tropical Easterlies are also overestimated by both models in the June-August season, as seen in Fig. 13. The annually averaged horizontal distribution of the zonal wind field at 200hPa, where the tropospheric jets

8

Fig. 12. Zonally averaged zonal wind for the months of Dec-Feb from CAM-SE, CAM-FV and NCEP.

Fig. 13. Zonally averaged zonal wind for the months of June-August, from CAM-SE, CAM-FV, and NCEP.

of the height field, where E(k) is defined as in Sec. 5). We define the resolved wave number as the value of k where the smoothing has reduced E(k) by 50% from its original unsmoothed value. By this measure, CAM-FV has the roughest topography, resolving k < 71, while CAMEUL and CAM-SE use significantly smoother topography, with k < 40 and k < 33 respectively. In the future we plan on using an improved remapping algorithm to generate topography data sets tailored for CAM-SE and with less smoothing. Both CAM-SE and CAM-FV match observed pressure vertical velocity through most of the atmosphere, but significantly overestimate (more negative) the upward motion at the ITCZ. From the difference plot between CAM-FV and CAM-SE (not shown), CAM-FV is about ∼ 3hPa/day more negative than CAM-SE throughout the ITCZ. This quantity is an important measure of the vertical transport of many thermodynamical quantities that determine cloud formation and precipitation. Unfortunately, the reduced bias in CAM-SE is not translated into a more significant improvement in the simulated precipitation. The surface wind stress over the ocean is an important momentum driver for the ocean from the atmosphere. The two models are globally 0.01N/m2 apart and slightly

closer to each other than values estimated from the European Remote Sensing Satellite scatterometer data (Stoffelen and Anderson 1997a,b), as shown in Fig. 16. Both models overestimate the wind stress around the Southern Hemisphere midlatitudes and to a lesser degree the North Atlantic storm tracks, an issue that has been present in the CAM version 3 where it is attributed to underestimated gravity wave drag in the mountain stress contribution to the balanced atmospheric momentum flux (Collins, W. D. et al. 2006). CAM-SE values are higher than CAM-FV and are significantly further from observations in that region, suggesting that the gravity wave drag is too weak in CAM-SE, due to CAM-SE’s use of smoother topography. Considering some of the polar climate features described above, such as the Northern Hemisphere sea level pressure (Fig. 8), the Southern Hemisphere surface wind stress over the ocean (Fig. 16) and the stratospheric jets (Figs. 12,13), we note that there does not appear to be any significant improvement due to CAM-SE’s lack of a polar filter. CAM-SE and CAM-FV have very similar biases in sea level pressure, with the CAM-SE bias slightly worse in the North Pacific. Similarly for the Southern Hemisphere wind stress, CAMSE and CAM-FV both have a large bias that is slightly larger in CAM-SE. The Northern Hemisphere polar night

9

Fig. 14. Zonal wind at 200hPa for CAM-SE, CAM-FV and NCEP.

Fig. 15. Zonally averaged vertical velocity.

jet is improved in CAM-SE, but experiments show this is not due to the lack of a polar filter but due to the top-ofmodel dissipation used by CAM-SE.

a. Meridional Migration of ITCZ

In Fig. 17, the climatology (1981 to 2000) of zonal mean precipitation is shown from GPCP observations, CAM-SE, and FV. The top panel includes all longitudes in the mean. Observations (Fig. 17-a1) show that the peak of the ITCZ migrates from south to north of the equator from about January to July and then migrates southward from about August to December. The peak is situated at around 8◦ S and 8◦ N in January and July, respectively. The ITCZ covers a latitudinal belt of about 20◦ , and extends up to 20◦ S and 20◦ N in January and July, respectively. A notable feature is that its northward migration is comparatively slower than the southward retreat; it takes 7 to 8 months going north and only 4 to 5 months going south. Also, the intensity of the ITCZ is greater in the northern hemisphere during Boreal summer. The average precipitation at the peaks is on the order of 10 mm/day in the northern hemisphere. Inspection of the ITCZ in CAM-SE (Fig. 17b1) indicates that the broad features are largely captured by the model. The peak occurs at around 8◦ S in January, migrates northward until July, and then proceeds to return south. The northern hemisphere has a greater in-

7. Transient Features The mean climate presented in the preceding section is composed of transient features that span a wide range of space and time scales. Although differences in dynamical cores are not apparent in the mean simulation they may be more apparent in transient statistics due to differences seen at the smallest scales in the KE spectra plots. In this section we assess some of these transient features in CAMSE and CAM-FV as compared to observations. We focus on the seasonal migration of the inter-tropical convergence zone (ITCZ) and zonal propagation. These features are especially important in the tropical belt and regulate most of the tropical climate variability, however they are difficult to simulate satisfactorily (Gates et al. 1999; Hack et al. 1998; Hurrell et al. 2006; Lin et al. 2006; Wu et al. 2003).

10

Fig. 17. Mean precipitation from GPCP observations, CAM-SE and CAM-FV. Zonally averaged over the globe (top panel) and over 60E–90E (bottom panel). ional migration for the Indian longitudes (60◦ E to 90◦ E) from observations, CAM-SE and CAM-FV. Over this region, there are two ITCZs, one around 5◦ S year-round and another around 20◦ N during the South Asian monsoon season of June through September. The secondary ITCZ develops abruptly over 5◦ S in April, moves northward, and reaches 20◦ N and then moves southward in August and September. The primary ITCZ is situated over the Indian Ocean, whereas the secondary occurs over the Indian subcontinent. CAM-SE captures the primary aspects of the regional ITCZ such as the double peaks and seasonal migration, however a spurious ITCZ is simulated over 0◦ N to 10◦ N during January through March, the secondary peak over the Indian subcontinent is overestimated, and spurious rainfall around 30◦ N is simulated by the model. In the spatial distribution of precipitation (Fig. 10), the model simulates excessive rainfall north of the Equatorial Eastern Arabian Sea during January through March, which is manifested as the spurious ITCZ during the same time over 0◦ N - 10◦ N The spurious rainfall around 30◦ N is due to the excessive rainfall over the Himalaya during the South Asian monsoon season. In general, CAM-SE captures the broad features though significant biases present in CAM-FV persist in SE. Some of these shortcomings are long-standing issues in climate modeling (Hack et al. 1998; Gates et al. 1999) and are associated with the physics parametrization (Wu et al. 2003). The ITCZ precipitation is a result of nonlinear interaction among dynamical, thermodynamical, and physical processes, and imperfections in any of them could lead to shortcomings in simulating this phenomena.

Fig. 16. Surface wind stress for CAM-SE, CAM-FV, and NCEP. tensity as observed and the southward progression is faster than the northward as in the observations. The shortcomings include an overestimation of the intensity, especially in the northern hemisphere, and a secondary ITCZ at around 5◦ N during January through April that is manifested in the form of stationary rainfall belts at around 5◦ N to 8◦ N that persist all year. The latitudinal spread of the ITCZ is 30◦ - 40◦ , which is wider than observations. It is noted that (not shown here) the overestimation in intensity during Boreal summer is primarily over the Bay of Bengal, Central Pacific, West coast of Mexico and Panama regions. The spurious ITCZ during Boreal winter exists over the Indian Ocean and West Pacific. The excess latitudinal extent is due to spurious rainfall over the Tibetan Plateau and South Pacific Convergence Zone during Boreal summer. Fig. 17-c1 shows the same zonal mean precipitation from CAM-FV, and displays similar biases. Because the Earth’s surface has zonal asymmetry, characteristics of the ITCZ vary regionally. For instance, behavior over the Indian region is different from that over African and American regions (Srinivasan and Smith 1996). Therefore regional zonally averaged precipitation is also examined. The bottom panel of Fig. 17 shows the merid-

11

bances in the wavenumber range of 1 to 10 and period from 10 to 20 days is the ER. It is seen from Fig. 18 that the models satisfactorily capture these waves, except an overestimation of variances around wavenumber 2 to 4 and 60 to 80 day period. It appears that the models simulate some kind of stationary waves with scales similar to MJO. The n=1 WIG is another symmetric component of the westward propagating waves with less than a 2.5 day time scale. In this regime, the variance is too weak in the models. Fig. 19 shows the anti-symmetric waves. The variance in the MRG waves is captured well by the models. However, in the EIG regime, the variances are feeble, and phase speeds are too slow, scaled to shallow equivalent depths. The n = 2 WIG is also weak in both models (not shown here).

b. Zonal Propagation

In the tropics, especially over the equatorial belt, convection occurs at various spatial and temporal scales and propagates primarily in the zonal direction. Within convective systems, cloud clusters interact to produce a hierarchy of collective motions and give rise to convectively coupled equatorial waves (CCEWs) that control a substantial fraction of tropical variability. The scales of these waves in the simulations are examined using the methodology of Wheeler and Kiladis (1999). Figures 18 and 19 show symmetric and anti-symmetric components of the normalized power spectra of the outgoing longwave radiation (OLR) averaged from 15◦ S to 15◦ N from CAM-SE, CAM-FV, and observations (Liebmann and Smith 1996). This normalization procedure removes a large portion of the systematic biases within the models, and more clearly displays the model disturbances with respect to their own climatological variances at each scale. The regions of wavenumberfrequency space defining the (meridional mode n = -1) Kelvin, n = 1 equatorial Rossby (ER), n = 0 mixed Rossbygravity (MRG), n = 0 eastward inertio-gravity (EIG), n = 1 westward inertio-gravity (n=1 WIG), n=2 westward inertio-gravity (n=2 WIG) and Madden Julian Oscillation (MJO) modes are similar to Wheeler and Kiladis (1999). The conventional dispersion curves of shallow water modes for equivalent depths of 12, 25, and 50m are shown in the figures. These waves are readily identified in the observational spectra, and as in Wheeler and Kiladis (1999), the spectral peaks match best with an equivalent depth of around 25 m. Some of these modes are captured well by the models, particularly the Kelvin, ER, MRG waves and MJO. The two models show similar patterns, namely prominent Kelvin, ER, and MRG waves, and weak WIG, EIG waves and MJO. However, there are some shortcomings in the simulations: weak Kelvin, WIG, and EIG waves and MJO and some differences between the two simulations. In the models, the Kelvin waves have most of their variance centers around 50m equivalent depth at lower wavenumbers and centers around 25m equivalent depth at higher wavenumbers, which corresponds to 22.1m/s and 15.6m/s respectively as seen in the observations. There is an increase in the speed of Kelvin waves at smaller scales. However, there is a lack of power in Kelvin modes, especially at higher wave numbers (above 4). The models simulate spurious variances at wavenumber 1 and period 20 to 30 days. The variance in FV is marginally greater, although the scales of the Kelvin waves are found to be largely similar in both the models. The MJO is the other eastward propagating mode with period 30 to 70 days and wavenumbers 1 to 5. The symmetric component of the MJO is seen in Fig. 18. Both the models show some variance within these bands. The symmetric component of the westward propagating distur-

8. Regional Climate: South Asian Monsoon Another important evaluation of CAM-SE is the ability to simulate the regional aspects of climate, particularly the remarkable South Asian monsoon. The South Asian monsoon occurs from June through September. Fig. 20 shows observed June to September (JJAS) averaged rainfall (top left panel) and 850 hPa wind climatology (top right panel) in the vicinity of the South Asian monsoon region. The climatology is based on GPCP rainfall data from 1979 to 2000 and ERA-40 data for 850 hPa wind. Some of the notable features of South Asian monsoon seen in the figure are the occurrence of maximum rainfall over the head of the Bay of Bengal, the extension up to an area south of the Equator between 80◦ E and 100◦ E, and a secondary maximum over the West coast of India. In the vicinity of the South Asian monsoon region and during the same period, there is substantial rainfall over the Western Pacific and almost no rainfall over Saudi Arabia. A crossequatorial flow and south equatorial easterlies exist in the lower troposphere (Fig. 20-a2). The Somali jet passes over the Indian Peninsula and penetrates up to around 120◦ E i.e. into the Western Pacific. Fig. 20, panel b1 and b2 show the simulated rainfall and wind over the same region at 850 hPa. The model is able to capture some of the features e.g. rainfall over Bay of Bengal, the Indian subcontinent, and the Somali jet. Shortcomings include overestimated rainfall over the Bay of Bengal, West coast of India, Western equatorial Indian Ocean, Himalaya, and underestimated rainfall over the Eastern equatorial Indian Ocean (West coast of maritime continents). Both CAM-SE and CAM-FV simulate unrealistic heavy rainfall over Saudi Arabia and an overly strong simulated Somali jet. The pattern correlation between observed and simulated rainfall with CAM-SE and CAM-FV over the region between 40◦ E to 120◦ E and 20◦ S to 40◦ N is 0.59 and 0.89 respectively (see top of Fig. 20 c1 and c2). It is notable that the correlation coefficient of rainfall and wind is improved to 0.68 and 0.94 respec-

12

Fig. 18. Wheeler-Kiladis diagrams showing the OLR power spectra (symmetric components) for observations, CAM-SE and CAM-FV. tively with CAM-SE (see top of Fig. 20 b1 and b2). The ITCZ has become more realistic, and the biases over Himalaya have been mitigated to some extent, which can be attributed to smoother orography in SE. The dry bias over the equatorial western Indian Ocean is also less severe. The low level southwesterly jet is more realistic in CAM-SE. Fig. 21 shows the meridional circulation over the South Asian monsoon region and in particular, a strong ascent over the Indian subcontinent accompanied with strong descent over the southern Indian Ocean and low-level inflow towards the land surface and upper level outflow towards the Indian Ocean. These form the regional Hadley cell, the broad features of which the CAM-SE and CAM-FV models largely capture although the circulation strength is overestimated. This is associated with an overestimation of the precipitation over the Indian subcontinent and the Bay of Bengal. Another shortcoming of the models is that the precipitation is overestimated over the Himalaya, which is due to the fact that they simulate overly strong ascending motion over the Tibetan plateau (however, the ascent over Himalaya in the lower troposphere is a plotting artifact caused by extrapolating the winds to pressures exceeding the local surface pressure). The presence of the biases in

both models again suggests that they may be related to deficiencies in the physics parameterizations. 9. Conclusions CAM-SE represents the successful integration of an unstructured grid spectral finite element dynamical core into the CAM. Despite the large numerical differences between the spectral-element and finite-volume dynamical cores in CAM, the climates produced by the models agree remarkably well over a wide range of means and transient features. CAM-SE and CAM-FV each have strengths and weaknesses when compared to observations, but these differences are small when compared to the overall biases present in both models, as expected, due to the many parametrized physical processes in CAM. CAM-SE provides increased scalability at high resolution, obtaining excellent simulation rates at both 0.25◦ and 0.125◦ resolution. CAM-SE high-order accuracy and low dissipation allows it to fully capture the Nastrom-Gage transition in the KE spectra at 0.125◦ resolution. The scientific and performance results give us confidence that CAM-SE is an effective approach for developing a high-resolution capability within the CCSM.

13

Fig. 19. As in Fig. 18, only showing the anti-symmetric components. Acknowledgments.

REFERENCES

The authors are grateful to Jim Hack for his assistance in tuning CAM-SE for energy balance and Nigel Wood for many constructive comments. We also acknowledge the CAM4 developers, including but not limited to members of the Atmospheric Model Working Group. Members of the Software Engineering Working Group were instrumental in the integration of HOMME into the CAM. The CCSM project is supported by the National Science Foundation and the Office of Science (BER) of the U.S. Department of Energy (DOE). K.E. and M.T. were supported by the DOE BER SciDAC grant 06-13194. This research used the Oak Ridge Leadership Computing Facilities at the Oak Ridge National Laboratory, supported by the Office of Science of the U.S. DOE under Contract No. DE-AC05-00OR22725, the Argonne Leadership Computing Facility at Argonne National Laboratory, supported by the Office of Science of the U.S. DOE under contract DE-AC02-06CH11357, and Sandia National Laboratories facilities, managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. DOE National Nuclear Security Administration under contract DE-AC0494AL85000.

Adams, J. C. and P. N. Swarztrauber, 1997: Spherepack 2.0: A model development facility. NCAR Technical Note Technical Report TN-436-STR, National Center for Atmospheric Research, Boulder, Colorado. Available from http://www.ucar.edu/library/collections/technotes/technotes.jsp. Adler, R. F. and Coauthors, 2003: Global precipitation climotology project (gpcp) monthly precipitation analysis (1979-present). J. Hydrometeor., 4, 1147–1167. Carpenter, I., R. Archibald, K. Evans, J. Larkin, P. Micikevicius, J. Rosinski, J. Schwarzmeier, and M. A. Taylor, 2011: Progress towards accelerating homme on hybrid multi-core systems. Int. J. High Perf. Comput. Appl., under review. Collins, W. D. et al., 2006: The Community Climate System Model Version 3 (CCSM3). J. Climate, 19, 2122– 2143. Craig, A., M. Vertenstein, and R. Jacob, 2011: A new flexible coupler for earth system modeling developed for 14

Fig. 20. Contour plots of mean simulated rainfall (left) and wind (right) over the Indian Ocean, from observations, CAM-SE and CAM-FV. The CAM-SE and CAM-FV pattern correlation coefficient (PCC) with observations is also given. CCSM4 and CESM1. Int. J. High Perf. Comput. Appl., under review.

Evans, K. J., M. A. Talyor, and J. B. Drake, 2010: Accuracy analysis of a spectral element atmospheric model using a fully implicit solution framework. Mon. Wea. Rev., 138, 3333–3341.

Dennis, J., A. Fournier, W. F. Spotz, A. St.-Cyr, M. A. Taylor, S. J. Thomas, and H. Tufo, 2005: High resolution mesh convergence properties and parallel efficiency of a spectral element atmospheric dynamical core. Int. J. High Perf. Comput. Appl., 19, 225–235.

Fournier, A., M. Taylor, and J. Tribbia, 2004: The spectral element atmosphere model (SEAM): High-resolution parallel computation and localized resolution of regional dynamics. Mon. Wea. Rev., 132, 726–748.

Dennis, J., et al., 2011: CAM-SE: A scalable spectral element dynamical core for the community atmosphere model. Int. J. High Perf. Comput. Appl., to appear.

Gates, W. L., et al., 1999: An overview of the results of the atmospheric model intercomparison project (amip i). Bull. Amer, Meteor. Soc., 81, 29–55.

Dennis, J. M., 2003: Partitioning with space-filling curves on the cubed-sphere. Proceedings of the Workshop on Massively Parallel Processing at IPDPS’03, nice, France.

Gent, P. R., et al., 2011: The Community Climate System Model version 4. J. Clim., 24, 4973–4991, doi:10.1175/ 2011JCLI4083.1.

Donner, L. J., et al., 2011: The dynamical core, physical parameterizations, and basic simulation characteristics of the atmospheric component AM3 of the GFDL global coupled model CM3. J. Clim., 24, 3484–3519, doi:10. 1175/2011JCLI3955.1.

Giraldo, F., 2005: Semi-implicit time-integrators for a scalable spectral element atmospheric model. Quart. J. Roy. Meteor. Soc., 131, 2431–2454. Giraldo, F. X. and T. E. Rosmond, 2004: A scalable spectral element eulerian atmospheric model (SEE-AM) for 15

Jones, P., 1999: First- and Second-Order conservative reampping schemes for grids in spherical coordinates. Mon. Wea. Rev., 127, 2204–2210. Kalnay, E., et al., 1996: The ncep/ncar 40-year reanalysis project. Bulletin of the American Meteorological Society, 77 (3), 437–471, doi:10.1175/1520-0477(1996)077h0437: TNYRPi2.0.CO;2. Khouider, B., A. St-Cyr, A. J. Majda, and J. Tribbia, 2010: MJO and convectively coupled waves in a coarse resolution GCM with a simple multicloud parametrization. Journal of the Atmospheric Sciences, 0 (0), null, doi:10.1175/2010JAS3443.1, URL http://journals. ametsoc.org/doi/abs/10.1175/2010JAS3443.1, http://journals.ametsoc.org/doi/pdf/10.1175/ 2010JAS3443.1. Kim, Y., F. Giraldo, M. Flatau, C. Liou, and M. Peng, 2008: A sensitivity study of the kelvin wave and the madden-julian oscillation in aqua-planet simulations by the naval research laboratory spectral element atmospheric model (NSEAM). J. Geophys. Res., 113.

Fig. 21. Time (JJAS) mean zonally (60E-120E) averaged meridional circulation from the observations, CAMSE, and CAM-FV NWP: Dynamical core tests. Mon. Wea. Rev., 132, 133– 153.

Kistler, R., et al., 2001: The NCEP-NCAR 50-Year Reanalysis: Monthly Means CD-ROM and Documentation. Bulletin of the American Meteorological Society, 82, 247–268, doi:10.1175/1520-0477(2001)082h0247: TNNYRMi2.3.CO;2.

Gleckler, P., 2004: The sceond phase of the atmospheric model intercomparison project (AMIP2):Toward innovative model diagnostics. WCRP/WGNE Workshop, Meteo-France, Toulouse, France, November 11-14, 2002.

Lauritzen, P., A.Mirin, J.Truesdale, K.Raeder, J.Anderson, J. Bacmeister, and R. Neale, 2011: Implementation of new diffusion/filtering operators in the cam-fv dynamical core. Int. J. High Perf. Comput. Appl., to appear.

Hack, J. J., J. T. Kiehl, and J. W. Hurrell, 1998: The hydrologic and thermodynamic characteristics of the NCAR CCM3. J. Climate, 11, 1179–1206. Hamilton, K., Y. O. Takahashi, and W. Ohfuchi, 2008: Mesoscale spectrum of atmospheric motions investigated in a very fine resolution global general circulation model. J. Geophys. Res., 113.

Lauritzen, P., C. Jablonowski, M. Taylor, and R. Nair, 2010: Rotated versions of the jablonowski steadystate and baroclinic wave test cases: A dynamical core intercomparison. J. Adv. Model. Earth Syst., 2, doi:10.3894/JAMES.2010.2.15, URL http://adv-model-earth-syst.org/index.php/ JAMES/article/view/26, art.#15, 34pp.

Hurrell, J. W., J. J. Hack, A. S. Phillips, J. Caron, and J. Yin, 2006: The dynamical simulation of the community atmosphere model version 3 (CAM3). J. Climate, 19, 2162–2183.

Liebmann, B. and C. A. Smith, 1996: Description of a complete (interpolated) outgoing longwave radiation dataset. Bull. Amer, Meteor. Soc., 77, 1275–1277.

Jablonowski, C. and D. Williamson, 2011: The pros and cons of diffusion, filters and fixers in atmospheric general circulation models. Numerical Techniques for Global Atmospheric Models, Springer Lecture Notes in Computational Science and Engineering, P. H. Lauritzen, C. Jablonowski, M. A. Taylor, and R. D. Nair., Eds., Springer, Berlin, Heidelberg, New York, Lecture Notes in Computational Science and Engineeering, Vol. 80, 381– 493.

Lin, J. L., et al., 2006: Tropical intraseasonal variability in 14 ipcc ar4 climate models. part i: Convective signals. J. Climate, 19, 2665–2290. Lin, S.-J., 2004: A vertically lagrangian finite-volume dynamical core for global models. Mon. Wea. Rev., 132, 2293–2397.

Jackson, C. S., M. K. Sen, G. Huerta, Y. Deng, and K. Bowman, 2008: Error reduction and convergence in climate prediction. J. Climate, 21, 6698–6709.

Lindborg, E., 1999: Can the atmospheric kinetic energy spectrum be explained by two-dimensional turbulence? Fluid Mech., 388, 259–288. 16

Loeb, N. G., B. A. Wielicki, D. R. Doelling, G. Smith, D. Keyes, S. Kato, N. Manalo-Smith, and T. Wong, 2009: Towards optimal closure of the earth’s top-of-atmosphere radiation budget. J. Climate, 22, 748–766, doi:10.1175/ 2008JCLI2637.1.

Putman, W. and S. Lin, 2007: Finite-volume transport on various cubed-sphere grids. J. Comput. Phys., 227, 55–78, doi:10.1016/j.jcp.2007.07.022. Putman, W. and S. Lin, 2009: A finite-volume dynamical core on the cubed-sphere grid. Numerical Modeling of Space Plasma Flows: Astronum-2008, Astronomical Society of the Pacific Conference Series, Vol. 406, 268276.

Loft, R., S. Thomas, and J. Dennis, 2001: Terascale spectral element dynamical core for atmospheric general circulation models. Proceedings of the ACM / IEEE Supercomputing SC’2001 conference.

Ranˇci´c, M., R. Purser, and F. Mesinger, 1996: A global shallow-water model using an expanded spherical cube: Gnomonic versus conformal coordinates. Q. J. R. Meteorol. Soc., 122, 959–982.

Mishra, S. K., M. A. Taylor, R. D. Nair, P. Lauritzen, H. M. Tufo, and J. J. Tribbia, 2011a: Evaluation of the HOMME dynamical core in the aqua-planet configuration of NCAR CAM4: Rainfall. J. Climate, 24, 4037– 4055, doi:10.1175/2011JCLI3860.1.

Rayner, N., P. Brohan, D. Parker, C. Folland, J. Kennedy, M. Vanicek, T. Ansell, and S. F. B. Tett, 2005: Improved analyses of changes and uncertainties in sea surface temperature measured in situ since the mid-nineteenth century: The hadsst2 dataset. J. Climate, 19, 446–469.

Mishra, S. K., M. A. Taylor, R. D. Nair, H. M. Tufo, and J. J. Tribbia, 2011b: Performance of the homme dynamical core in the aqua-planet configuration of ncar cam4: equatorial waves. Annales Geophysicae, 29 (2), 221–227, doi:10.5194/angeo-29-221-2011, URL http:// www.ann-geophys.net/29/221/2011/.

Ringler, T. D., D. Jacobsen, M. Gunzburger, L. Ju, M. Duda, and W. Skamarock, 2011: Exploring a multiresolution modeling approach within the shallow-water equations. Mon. Wea. Rev., 139, 3348–3368, doi:10. 1175/MWR-D-10-05049.1.

Murphy, D., S. Solomon, R. Portman, K. Rosenlof, P. Forster, and T. Wong, 2009: An observationally based energy balance for the earth since 1950. J. Geophys. Res., 114, D17 107, doi:10.1029/2009JD010105.

Rossow, W. and R. Schiffer, 1999: Advances in understanding clouds from ISCCP. Bull. Amer, Meteor. Soc., 80, 2261–2287.

Nair, R. D., 2009: Diffusion experiments with a global discontinuous galerkin shallow water model. Mon. Wea. Rev., 137, 3339–3350.

Sadourny, R., 1972: Conservative finite-difference approximations of the primitive equations on quasi-uniform spherical grids. Mon. Wea. Rev., 100 (2), 136–144.

Nastrom, G. and K. S. Gage, 1985: A climatology of atmospheric wavenumber spectra of wind and temperature observed by commercial aircraft. J. Atmos. Sci., 42, 950–960.

Skamarock, W., 2004: Evaluating mesoscale NWP models using kinetic energy spectra. Mon. Wea. Rev., 132, 3019–3032. Skamarock, W., 2011: Kinetic energy spectra and model filters. Numerical Techniques for Global Atmospheric Models, Springer Lecture Notes in Computational Science and Engineering, P. H. Lauritzen, C. Jablonowski, M. A. Taylor, and R. D. Nair., Eds., Springer, Berlin, Heidelberg, New York, Lecture Notes in Computational Science and Engineeering, Vol. 80, 495–512.

Neale, R., J. Richter, S. Park, P. Lauritzen, S. J. Vavrus, P. Rasch, and M. Zhang, 2011: The mean climate of the Community Atmosphere Model (CAM4) in forced SST and fully coupled experiments. J. Climate, under review. Neale, R., et al., 2010: Description of the Community Atmosphere Model (CAM 4.0). NCAR Technical Note, TN-X+STR.

Srinivasan, J. and G. L. Smith, 1996: Meridional migration of tropical convergence zones. Journal of Applied Meteorology, 35 (8), 1189–1202, doi:10.1175/1520-0450(1996) 035h1189:MMOTCZi2.0.CO;2.

Neale, R. B. and B. J. Hoskins, 2000a: A standard test case for AGCMs including their physical parametrizations: I: The proposal. Atmos. Sci. Lett., 1, 101–107, doi:10. 1006/asle.2000.0022.

St.-Cyr, A., C. Jablonowski, J. M. Dennis, H. M. Tufo, and S. J. Thomas, 2008: A comparison of two shallow water models with non-conforming adaptive grids. Mon. Wea. Rev., 136, 1898–1922.

Neale, R. B. and B. J. Hoskins, 2000b: A standard test case for AGCMs including their physical parametrizations: II: Results for the Met Office Model. Atmos. Sci. Lett., 1, 108–114, doi:10.1006/asle.2000.0024.

17

Stoffelen, A. and D. Anderson, 1997a: Scatterometer data interpretation: Estimation and validation of the transfer function CMOD4. J. Geophys. Res., 102(C3), 5767– 5780, doi:10.1029/96JC02860.

Trenberth, K. and J. M. Caron, 2001: Estimates of meridional atmosphere and ocean heat transports. J. Climate, 14, 3433–3443. Wang, H., J. J. Tribbia, F. Baer, A. Fournier, and M. A. Taylor, 2007: A spectral element version of CAM2. Monthly Weather Review, 135, 3825–3840, doi:10.1175/ 2007MWR2058.1.

Stoffelen, A. and D. Anderson, 1997b: Scatterometer data interpretation: Measurement space and inversion. J. Atmos. Oceanic Technol., 14, 1298–1313. Stratton, R. A., 1999: A high resolution amip integration using the hadley centre model hadam2b. Climate Dynamics, 15, 9–28.

Wheeler, M. and G. N. Kiladis, 1999: Convectively coupled equatorial waves: Analysis of clouds and temperature in the wavenumberfrequency domain. J. Atmos. Sci., 56, 374–399.

Taylor, K., 2001: Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res., 106, 7183–7192, doi:doi:10.1029/2000JD900719.

Williamson, D. L., 2002: Time-split versus process-split coupling of parameterization and dynamical core. Mon. Wea. Rev., 130, 2024–2041.

Taylor, M., 2011: Conservation of mass and energy for the moist atmospheric primitive equations on unstructured grids. Numerical Techniques for Global Atmospheric Models, P. H. Lauritzen, C. Jablonowski, M. A. Taylor, and R. D. Nair., Eds., Springer, Berlin, Heidelberg, New York, Lecture Notes in Computational Science and Engineeering, Vol. 80.

Williamson, D. L., 2008a: Convergence of aqua-planet simulations with increasing resolution in the community atmospheric model, version 3. Tellus, 60, 848–862. Williamson, D. L., 2008b: Equivalent finite volume and spectral transform horizontal resolutions established for aqua-planet simulations. Tellus, 60, 839–847.

Taylor, M., R. Loft, and J. Tribbia, 1998: Performance of a spectral element atmospheric model (SEAM) on the HP Exemplar SPP2000. Tech. Rep. TN–439+EDD, NCAR, 16 pp.

Williamson, D. L., J. Kiehl, and J. Hack, 1995: Climate sensitivity of the NCAR Community Climate Model (CCSM2) to horizontal resolution. Climate Dynamics, 11, 377–397.

Taylor, M., J. Tribbia, and M. Iskandarani, 1997: The spectral element method for the shallow water equations on the sphere. J. Comput. Phys., 130, 92–108.

Worley, P. H., A. P. Craig, J. M. Dennis, A. A. Mirin, M. A. Taylor, and M. Vertenstein, 2011: Performance and performance engineering of the community earth system model. Proceedings of the ACM / IEEE Supercomputing SC’2011 conference.

Taylor, M. A., J. Edwards, and A. St.-Cyr, 2008: Petascale atmospheric models for the community climate system model: New developments and evaluation of scalable dynamical cores. J. Phys. Conf. Ser., 125 (012023), doi: 10.1088/1742-6596/125/1/012023.

Worley, P. H. and J. B. Drake, 2005: Performance portability in the physical parameterizations of the community atmospheric model. International Journal of High Performance Computing Applications, 19, 187 – 201.

Taylor, M. A., J. Edwards, S. Thomas, and R. Nair, 2007: A mass and energy conserving spectral element atmospheric dynamical core on the cubed-sphere grid. J. Phys. Conf. Ser., 78 (012074), doi:10.1088/1742-6596/ 78/1/012074.

Wu, X., X.-Z. Liang, and G. J. Zhang, 2003: Seasonal migration of ITCZ precipitation across the equator: Why cant GCMs simulate it? Geophys. Res. Lett., 30, 1824, doi:10.1029/2003GL017198.

Taylor, M. A. and A. Fournier, 2010: A compatible and conservative spectral element method on unstructured grids. J. Comput. Phys., 229, 5879– 5895, doi:10.1016/ j.jcp.2010.04.008. Thomas, S. and R. Loft, 2002: Semi-implicit spectral element atmosphere model. Journal of Scientific Computing, 17, 339–350. Thomas, S. and R. Loft, 2005: The NCAR spectral element climate dynamical core: Semi-implicit Eulerian formulation. J. Sci. Comput., 25, 307–322. 18