Scientific computing in the age of complexity

feature

Scientific Computing in the Age of Complexity Climate modeling has come a long way since von Neumann declared it a problem too hard for pencil and paper, but tailor-made for the new digital computers. As the models and computers both evolve toward ever-greater complexity, they are changing our notions of digital simulation itself. By V. Balaji DOI: 10.1145/2425676.2425684

T

THE HEROIC PERIOD: CLIMATE SCIENCE AND COMPUTING COME OF AGE There was once, of course, nothing but scientific computing. When John von Neumann invented modern digital computing at the Institute for Advanced Study back in the middle of the 20th century, the only applications he had in mind were scientific: Problems in physics and chemistry and engineering that required computations that could not be measured in blackboard widths. Not even the giants back then, whose shoulders we now stand on, imagined that what they were building would become mundane objects of everyday life—used by surgeons to operate robotic scalpels from thousands of miles away, or for citizens to record and broadcast acts of state repression of 12

peaceful political protest. Hard science problems were what they were after. My own lab, the Geophysical Fluid Dynamics Laboratory (GFDL), can trace its ancestry back to that dawn of computing. von Neumann explicitly listed weather and climate as computational problems too hard for pencil and paper, but quite suitable for ENIAC and its descendants. It was at his behest that the U.S. established centers for the study of weather and climate in Washington; one of which, the GFDL, found its way back a few years later to its roots in Princeton—near von Neumann’s Institute—where it has been for the last half century. Joe Smagorinsky, who served as GFDL’ s director for its first decades, was von Neumann’s colleague at the Institute.

It was at the GFDL that Syukuro Manabe, Kirk Bryan, and their colleagues pioneered the study of the Earth system using computers. The models they used were not based on empirical or statistical fits. Instead, it was a bold, even hubristic, attempt to solve more or less from first principles the time evolution of the Earth system—a complex evolving mixture of fluids and chemicals in a very thin layer atop a wobbling, spinning sphere with an unstable surface and a molten interior, zooming through space in a field of extraterrestrial photons at all wavelengths. Between sea and sky lay that thin layer of green scuzz that contained all the known life in the universe, which itself was capable of affecting the state of the whole system. XRDS • spring 2013 • Vol.19 • No.3

Image courtesy of Thomas Delworth and Tony Rosati, NOAA/GFDL.

he history of scientific computing told from the point of view of climate modeling is a story of dramatic increases in the complexity of the science—our understanding of the many linked processes that govern the Earth system—along with similarly radical and disruptive changes in the complexity of computing: in architectures, algorithms, and software. These transitions are bringing us to a new era that will reshape how we think about digital simulation itself.

XRDS • spring 2013 • Vol.19 • No.3

13

Robert Oppenheimer and John von Neumann in front of (part of) ENIAC.

The Lorenz Butterfly, considered one of the most beautiful images of modern physics, whose elegant lines also suggest the powerful metaphor of the “butterfly effect.”

14

These were solved by applying Newton’s laws to rotating spherical fluid layers, subject to solar heating at the Equator and losing heat at the poles. The response required to move that energy polewards led to atmospheric circulations that explained the persistent rain that gave rise to forests around the Equator, the dry desert zones in a band around the globe in either hemisphere near the Tropics, and the sweeping storm systems that periodically came through the higher latitudes in winter. A similar computation in the oceans explained the wellknown marine currents and a giant circulation system that connected all the seas, pole to pole. The original computers had to be spoken to in “machine language”: a first major breakthrough in the co-evolution of computational science and computers was the invention of highlevel languages. Fortran (a contraction of “formula translation,” pointing to its roots in scientific computing, and still the mostly widely used language in climate modeling) allowed scientists to express their equations in quite human-readable terms, which would then be translated into the relatively opaque language of machines. In this, the first era of computational climate science, scientists were able to write their own programs and execute them on the biggest computers then available. Using these codes, Manabe and colleagues first showed that imposing increased CO2 levels upon the Earth system led irrevocably to the warming of the planet. It became apparent that the oceans played a leading role in its long-term thermal balance. This coupled system too was first modeled at GFDL, a computational tour de force recognized by Nature as a milestone in scientific computing, alongside handheld calculators and the World Wide Web. Along the way, puzzling and unexpected results popped out. Lorenz noticed very small errors in computations or data could rapidly grow in amplitude and overwhelm the system under study, completely altering the solution. He was able to distill this down to a very simple set of equations, which demonstrate that the future state of a system could be uncomputXRDS • spring 2013 • Vol.19 • No.3

lan Richards photographer. From the Shelby White and Leon Levy Archives Center, Institute for Advanced Study, Princeton, NJ, USA. Image source; http://en.wikipedia.org/wiki/File:Lorenz_attractor_yb.svg

feature

able even though its physics is fully known; there are limits to the predictability of even a fully deterministic physical system. This discovery of chaos led to one of the most beautiful images of modern physics, whose elegant lines also suggest the powerful metaphor of the “butterfly effect,” where “a flap of a butterfly’s wing in the Amazon can, given enough time, give rise to a tornado in Texas” (the vision is often attributed to Smagorinsky). It showed the Lorenz limit to any forecast: Since the initial conditions cannot be known with infinite precision, the solution of a chaotic system will degrade over time to become no more useful than a coin toss. For planetary weather, this limit is just a few days. Yet it was becoming clear that there were long-term variations that needed to be understood. Why was the monsoon over India strong one year and feeble the next? Was this due to some changes externally forced or an internal mode of the system? Strictly periodic modes such as the diurnal cycle or the passing of the seasons, could be easily understood and traced to orbital phenomena of the Earth’s rotation and solar revolution. But there was no easily detectable pattern to the monsoons, on which India’s agriculture is dependent. Global general circulation models were able to show such aperiodic patterns, purely as a result of the internal variability of the system. Over some years, George Philander at Princeton and others, elucidated the mechanisms underlying the El Niño Southern Oscillation (ENSO), an aperiodic fluctuation in Pacific Ocean temperatures that could be linked to weather patterns all over the world. The models showed there was some predictability at these timescales: While there is no chance of predicting a particular day’s weather months ahead, you could in fact, with some skill, predict if the next season would be rainier than usual or unseasonably warm. Just as Lorenz with his three-variable model of chaos, El Niño could be explained using simple physical models, such as a delayed oscillator. This pattern of science—showing something working in a computationally challenging first principles model of the planet, XRDS • spring 2013 • Vol.19 • No.3

When John von Neumann invented modern digital computing at the Institute for Advanced Study back in the middle of the 20th century, the only applications he had in mind were scientific. and then explaining the workings of the full system using a physically intuitive system with very few degrees of freedom—is a constant theme in the field. We think of this as the “hierarchy of models” approach, moving back and forth between simple and complex models of the Earth system, or individual phenomena within it.

ADOLESCENCE: CLIMATE MODELS IN THE SERVICE OF POLICY; PARALLEL COMPUTING By the 1990s, the basic physics governing the evolution of the planetary climate were quite well understood. While we still could not run global models of the resolution needed to resolve key physical processes, such as clouds, models that used approximate representations of the aggregate behavior of many clouds were sufficiently good that we could use them to run projections of future climate. Predictions were not possible, as had already been shown, yet we could run many instances of a model—an ensemble —to describe a range of possible different outcomes and their probability distribution under various scenarios of global industrial and agricultural policy (which would prescribe how much of CO2 and other pollutants would enter the climate system). The results from such models were sufficiently alarming that the world began to take serious notice of how humans were directly interfering with a complex system that was the only one in the universe known to

support carbon-based life forms. It soon became apparent more and more processes and feedbacks needed to be included to extract the information needed for policy. We needed to model not just circulations in the ocean and atmosphere, but a complex chemistry involving trace constituents. The dynamics of ecosystems on land and at sea—sometimes including individual species, such as the pine bark beetle—played important roles. The models were clearly beyond any single scientist’s mastery: No one could know all of the science that went into different components of the Earth system. Model building became “Big Science”; large groups of scientists each specializing in a different aspect of the system teaming up to build comprehensive models. This transition to collaborative science was accompanied by a disruptive change in computing. The specialized machines of the 1970s and 1980s engineered and built by Seymour Cray and others had been tailored for scientific computing: They were easy for scientists to program and run. But computing was now marching to the tune of a different drummer. The industry was no longer driven by the lumbering giants of the “big iron in a cold room” era of computing: It was now focused on the newly imagined world of cheap, ubiquitous, personal computing aimed at a market of individual citizen consumers. Big Science no longer controlled the marketplace and would have to figure out how to come along for the ride. This led to a dramatic transition in the way climate models were built. Software engineering became a key issue: How do you couple a land model built by one group of scientists to an atmosphere built by a different team, to an ocean built by yet a third, living on another continent, all while respecting numerical constraints on coupling, such as the conservation of mass and energy across the whole system? Extracting performance from chips not primarily built for number crunching also proved to be a daunting task. For the first time, scientists turned to professional programmers to help saddle their beasts. The use of software frameworks, of 15

Sea surface temperature (SST) simulation from GFDL’s high resolution coupled atmosphere-ocean model. More data visualizations are available at http://www.gfdl.noaa.gov/visualization

which the GFDL Flexible Modeling System is an early example, was pioneered in this period. They typically consisted of an infrastructure, providing highlevel class libraries in which scientists could express their algorithms, and a superstructure, allowing different codes for different system components—land, atmosphere, and so on— to be coupled with the required numerical and algorithmic constraints. Between these two layers came the scientific code. The infrastructure was adapted to working on large clusters: parallel and distributed-memory clusters being the prevailing architectural model of that era. These were essentially systems cobbled together out of the same processors that powered consumer electronics, tightly clustered on a network. Frameworks were soon prevalent across the big modeling centers, and there were community standard frameworks built, such as the Earth System Modeling Framework. Despite the difficulties of working with the new hardware, these software advances enabled a huge surge in the computing power harnessed for climate science. This surge was motivated in part by 16

the use of ensembles to capture uncertainty. Consider the forecast problem: An initial value problem where we are attempting to predict the future state of the system given its history. As described earlier, in the case of weather forecasting, this is a prediction over a time scale of a few days. The initial state is only approximately known, and what data that exists is corrupted by noise. Initial-value ensembles attempt to reduce dependency on error in the initial conditions by running several realizations of a single model over a

In this, the first era of computational climate science, scientists were able to write their own programs and execute them on the biggest computers then available.

probability distribution of initial values (data + a probability distribution of error). This approach has been shown to be able to forecast extreme events and severe weather better than individual runs. Climate modeling is different from weather modeling (recall the aphorism: Climate is what you expect, weather is what you get). It is a boundary-value problem where we are attempting to enumerate the set of possible states of the system for some given values of external parameters (e.g anthropogenic CO2). Some of these external parameters are poorly constrained by data. A large body of literature attempts to understand the response of an individual climate model to changes in the external parameters. These perturbed-parameter ensembles can be used to sample uncertainty in the values of external parameters, as well as study the response of the model to climate change. An extreme example of this is the climateprediction.net project modeled on SETI@Home, which uses volunteer computing—individuals donating idle time on their home computers—to perform a dense sampling of external parameter space using a climate model running as a screensaver on PCs worldwide. These approaches both focus on uncertainties associated with a single model. An even deeper approach also attempts to sample across uncertainties in our understanding—known as structural uncertainty. There are some processes—clouds, for example— whose representation in models is still in question. Different modeling groups around the world use differing representations. By coordinating and running the same experiment and pooling results, we arrive at a method for the comparative study of models, the multi-model ensemble (MME). These so-called model intercomparison projects or MIPs have become standard operating procedure in arriving at both consensus and uncertainty estimates of climate change. Results from MIPs also show intriguing results. For instance, the average model (the arithmetic mean result across all models in an MME) shows better skill on many metrics, than any individual model. Such analyses are the basis for XRDS • spring 2013 • Vol.19 • No.3

Image courtesy of Thomas Delworth and Tony Rosati, NOAA/GFDL.

feature

policy documents such as the Intergovermental Panel on Climate Change (IPCC) Assessment Reports, which are periodic accounts of the state of the science, released every few years. The IPCC won a Nobel Peace Prize in 2007 (along with Al Gore) for its efforts in informing the world community of the causes and consequences of climate change. The global data network for distributing climate model projections, along with the network for collecting data from atmospheric and ocean sensors, and satellites, is now part of the “vast machine” that is today’s global weather and climate science enterprise.

The world began to take serious notice of how humans were directly interfering with a complex system that was the only one in the universe known to support carbon-based life forms.

THE AGE OF UNCERTAINTY The first age of computational climate science, done on specialized hardware, led to fundamental breakthroughs in our understanding of dynamical systems, such as the unsettling knowledge that even completely deterministic systems may not be predictable. In the second age, the widespread availability of cheap computing made it possible to deal with this indeterminism by running many, many instances of a model, and capturing its complete range of behaviors, including low-probability events of high impact. Further breakthroughs in the second age focus on ways to overcome gaps in our knowledge, structural uncertainty, these are done by sampling across many models. So long as the models are sufficiently different from each other, their biases will be uncorrelated, and will cancel each other out in the average model. This approach is well within the spirit of today’s age, where many algorithms for extracting information from public data uses the “wisdom of crowds.” An even clearer parallel is with the rise of the “poll of polls” in election prediction; particularly in the case of Nate Silver, arguably the first psephologist to achieve rock star status, with his uncannily accurate predictions of the 2012 U.S. elections. That approach—a judicious weighted average across individual polls—is very close to the averagemodel view in interpreting MMEs. Yet there remain weaknesses in this approach. Averaging across mod-

els may seem to show greater skill, but does not in itself advance understanding. This is a somewhat uneasy situation. Climate models are evaluated for their skill in simulating the known climate record of the last 100-odd years, initialized in, say 1850. We often see cases of models that are indistinguishable in skill against past climate, yet make opposite predictions about the future evolution of some climate variable of interest, say rainfall over the Sahel region of Africa. But comprehensive models are not good tools for genuine physical insight. Complex systems, we know today, have a rich range of emergent behavior that may be bewildering and counter-intuitive. It is clear that today’s models, which may simulate the interactions of hundreds or even thousands of physical, chemical, and biological constituents and species are well past any threshold of immediate intuitive grasp. For that we must return to the hierarchy of models, where we use models with greatly reduced degrees of freedom to isolate and understand individual phenomena. A colleague compares this to the various animals used in genetics research for example: One may think of some simple model as the climate’s E. coli, another, more complex, as C. elegans, a third as the mouse, and so on. Once we convince ourselves that our basic physics is cor-

XRDS • spring 2013 • Vol.19 • No.3

rect, we may be more inclined to be comfortable with, and even revel in, counter-intuitive results from a full model, and be willing to give credence to the wisdom of the ensemble. Further developing the biological analogy, perhaps we will use our software frameworks to facilitate crosspollination and hybridization between models: Attempt to understand divergent results between models by exchanging components and narrowing the list of points where structural uncertainty resides. We will still be left with the natural indeterminism of chaotic systems. Once again, it is the spirit of the age: Computing is moving in a similar direction. Moore’s law—where miniaturization and processor frequency increase exponentially in time—has run its course. Future systems are all built on the premise of increasing on-chip concurrency, with no increase in clock speed. This has led to an explosion of functional units on a chip, which potentially tick forward in an asynchronous fashion. This is far from the exactly repeatable von Neumann model of computing: Here the order of operations on a chip cannot be guaranteed, and thus neither can the results of a computation, since floating-point arithmetic is order-dependent. This is an unsettling vision, but scientific computing may have no option but to embrace the chaos. Like cells in a biological experiment, an individual run of a computation may no longer be exactly reproducible: only the ensemble—the cell line—is. In silico experimentation begins to be more like in vitro experimentation. Models exchange components; adaptive algorithms tune themselves to the underlying hardware. Codes are grown like cultures on a digital petri dish. The model hierarchy underpins our understanding of the basic processes, and when we assemble them into complex ones, we learn to trust them like we trust life itself. Biography Dr. V. Balaji works at the porous boundary of science and software, Head of Modeling Systems at Princeton University’s Cooperative Institute for Climate Sciences, and NOAA’s Geophysical Fluid Dynamics Laboratory.

© 2013 ACM 1529-4972/13/03 $15.00

17