general requirements for simulation models in waste ... - CiteSeerX

WM’03 Conference, February 23-27, 2003, Tucson, AZ

GENERAL REQUIREMENTS FOR SIMULATION MODELS IN WASTE MANAGEMENT Ian Miller, Rick Kossik, Charlie Voss Golder Associates Inc 200 - 18300 Union Hill Road NE, Redmond, WA 98052 ABSTRACT Most waste management activities are decided upon and carried out in a public or semi-public arena, typically involving the waste management organization, one or more regulators, and often other stakeholders and members of the public. In these environments, simulation modeling can be a powerful tool in reaching a consensus on the best path forward, but only if the models that are developed are understood and accepted by all of the parties involved. These requirements for understanding and acceptance of the models constrain the appropriate software and model development procedures that are employed. This paper discusses requirements for both simulation software and for the models that are developed using the software. Requirements for the software include transparency, accessibility, flexibility, extensibility, quality assurance, ability to do discrete and/or continuous simulation, and efficiency. Requirements for the models that are developed include traceability, transparency, credibility/validity, and quality control. The paper discusses these requirements with specific reference to the requirements for performance assessment models that are used for predicting the long-term safety of waste disposal facilities, such as the proposed Yucca Mountain repository. INTRODUCTION Simulation models are generally used to investigate alternative design or operational alternatives prior to a decision point. Well-constructed, credible simulation models can be a cost-effective way to explore the risks and benefits of alternative decisions without incurring the time and expenses of experimental or prototype projects. As computer hardware and software capabilities have advanced, simulation models have become an accepted, and even required, part of the process of developing and evaluating waste management alternatives. Most waste management activities are decided upon and carried out in a public or semi-public arena, typically involving the waste management organization, one or more regulators, and often other stakeholders and members of the public. In these environments, simulation modeling can be a powerful tool in reaching a consensus on the best path forward, but only if the models that are developed are understood and accepted by all of the parties involved. These requirements for understanding and acceptance of the models constrain the appropriate software and model development procedures that are employed.

INTRODUCTION TO SIMULATION MODELS By simulation models we refer to computer-based models that simulate the dynamic evolution of a real or proposed system1,2. In general, the term simulation is only used for models of complex, multi-component systems and not for finite element or finite difference models of specific physical processes. With the maturation of powerful personal computers, a wide range of high-quality simulation software packages has become available3,4. These packages are suited for a wide variety of applications. Some packages are quite specialized, and some are more general in their applicability. The available simulation models can be roughly divided into four categories: • Spreadsheet-based models. • Discrete-event models. • System-dynamics models. • Hybrid models. Spreadsheet-based models are undoubtedly the most widely used, with Microsoft Excel being the predominant software package. These models typically use columns in the spreadsheet to represent the system’s state variables at a point in time. The dynamic evolution of the model is computed by making each row a function of the row to its left, thus representing a discrete time step. Spreadsheet-based models are simple to construct, and can be enhanced by the use of probabilistic add-ins such as Crystal Ball and @Risk. However, they have difficulty dealing with systems whose properties change dynamically, random events, and feedback links. Discrete-event models such as Pro-Model, Arena and Witness rely on a transaction-based approach to modeling specific changes to the system. These models consist of entities (units of traffic), resources (elements that service entities), and control elements (elements that determine the states of the entities and resources). Discrete simulators are generally designed for simulating detailed processes such as call centers, factory operations, and shipping facilities. Discrete-event models typically handle random stochastic processes well, but are not designed to incorporate uncertainty about internal or external attributes of the system. By and large they are most useful for simulating the performance of a fully-designed system, and are seldom used during the conceptual design phases. System-dynamics models such as Stella, Vensim and Powersim are generally based on the stock and flow approach5 developed by Professor Jay W. Forrester at MIT in the early 1960’s. Models based on system dynamics use three types of elements: stocks, which represent quantities of material, flows, which represent the flows of materials between stocks, and converters, which define the properties of the stocks and flows as a function of the system status. These models step through the simulation time frame using fixed time steps, and typically use Runge-Kutta or other sophisticated algorithms for integrating the system equations. System-dynamics models are a flexible and powerful approach to simulate continuously-varying systems, such as general business systems and simple engineering and scientific systems. Typically they do not incorporate uncertainty about internal or external attributes of the system, and are incapable of representing discrete events.

Hybrid models such as GoldSim or Extend integrate discrete-event modeling capabilities with systemdynamics type continuous simulation. Typically these models are more flexible than the discreteevent or system-dynamics models, as they are not designed for a specific class of problems. Simulation Model Applications in Waste Management Simulation models can be applied to numerous aspects of waste management, such as: • Logistics simulation, considering stores of waste material, transportation and/or processing, and the equipment and supplies that are required. • Simulation of single- and multi-machine processes, for example processes where wastes are processed or packaged. • Simulation of the environmental fate and transport of waste constituents, leading to the possible exposure of receptors (both human and ecological) to the contaminants, and the resulting risk of health effects. • Simulation of the costs and schedules for a waste management project or, at a higher level, for an entire waste management program. Issues Addressed by Simulation Models Simulation models can be used to address different issues involved in waste management, for example: • Models can be used to test the physical feasibility of a proposed system: geometries and interactions of machines, timings of tasks, stresses on components, etc. • Simulations can deal with random (stochastic) risk factors, such as weather events, earthquakes, or even human activities (terrorism, mistakes, etc.). These types of simulations support the evaluation of risk levels for a system, and can assist in minimizing or mitigating the risks. • Simulations can also deal with the effects of uncertainty in processes or parameters, in order to develop a second type of confidence bounds around predictions. These types of simulations help prioritize R&D activities, and establish the point at which a proposed system is sufficiently well-understood (and safe) to proceed. They can also help to quantify the level of programmatic or cost risk that a particular waste management system may have.

REQUIREMENTS FOR SIMULATION MODELS While simulation models can make a valuable contribution to waste management issues, they can have a negative impact if they fail to meet the key measures of understanding and acceptance by the stakeholders. Waste management decisions are fraught with conflict and differing risk perceptions between the parties, and simulation models that are not accepted can lead to even deeper divisions between the parties. With these challenges in mind, the authors have developed a number of guidelines for successful simulation models, which are presented below in the form of requirements for both the software that is used and for the models that are developed. In general, simulation models consist of a software system and a model. It is important to differentiate between the software and the model: • The software is the developed or acquired computer code that will perform the simulation calculations. • The model has several components: o The ‘conceptual model’ that is the agreed-upon representation of how the system works, o The ‘computer model’ that is the coded representation of the conceptual model, built using the software, and o The model data, which may be entered directly into the software or accessed by the software via a database or spreadsheet. Requirements for Simulation Software Requirements for the software include: • Transparency, which allows a user or reviewer of a model built using the software to fully understand how it works. • Accessibility, which means that the software runs on a readily-available computer platform (e.g., Microsoft Windows) and is available at a reasonable price. • Flexibility, which means that the software can support models of many kinds of systems, for example combining financial, environmental, and logistics issues in a single simulation. • Extensibility, which means that the software can incorporate user-provided model components, such as spreadsheets and user-provided software modules. • Quality assurance (verification), which provides confidence that the software will produce the correct answers. • Ability to do discrete and/or continuous simulation, to deal with model components that evolve continuously over time and/or components that evolve in discrete stages. Depending on the application either continuous, discrete, or combined continuous/discrete modeling may be required. • Efficiency, so that simulations can be carried out in a timely manner without excessive investments in computer hardware. Requirements for Simulation Models Requirements for the model itself are distinct from those for the software, and usually include: • Traceability, meaning that the model assumptions and data were derived from welldocumented processes with appropriate quality control. • Transparency, meaning that the entire model is documented and comprehensible by all the parties.

• •

Credibility, meaning that the development of the conceptual model and the supporting data have been reviewed by technically qualified independent reviewers, and agreed to by all of the key stakeholders. Quality control, meaning that the model and data as implemented using the software have been developed and reviewed under appropriate quality controls, to ensure that the model that was built is the model that was intended.

REQUIREMENTS FOR PERFORMANCE ASSESSMENT (PA) SOFTWARE While the requirements summarized above are general and apply to all types of simulation, their implementation may differ in different application areas. The following sections discuss the application of the requirements in the area of performance assessment modeling, which involves simulating the longterm performance of a real or proposed waste disposal facility. Performance assessment modeling is also commonly used to evaluate the safety of a ‘leave in place’ option for sites of existing environmental contamination. The examples presented are primarily taken from the models being developed to support the license application for the proposed Yucca Mountain spent fuel and high level waste repository. The Yucca Mountain Project uses the GoldSim software package, developed by the authors, as the primary basis for its long-term simulations of repository performance. During the development of GoldSim, many of its design features were specifically intended to support the Yucca Mountain application. Requirement for Transparency The Yucca Mountain PA model is probably one of the most complex probabilistic models ever built, and in order to make it transparent a number of innovations were required. Some of the specific software requirements in this area include the following: • The model uses a visual, object-oriented interface where each component of the model has a visual and textual representation, as illustrated in Figure 1. • The model can be organized hierarchically into systems and subsystems, and accessed using a browse-tree, in order to make the logical relationships of its components more transparent. • The properties of each model component can be directly viewed by double-clicking on the object. • Linkages between model components are also visible. • Dependency relationships throughout the model are readily displayed in the form of dependencytrees. • All objects in the model can have a description and notes attached by the model builder, in order to explain the logical basis and provide references for the design of the object. • The model builder can also add hyperlinks pointing to external reference documents, including Internet sites, Word documents, and PowerPoint presentations, in order to provide additional elaboration and documentation. • External documents which describe aspects of the model can embed hyperlinks that will automatically open the PA model and jump to the referenced component. • All model results, including charts and tables, are contained within the model and directly accessible by double-clicking.

Fig. 1. Example of Visual, Object-Oriented Software Interface Requirement for Accessibility The model software is commercially available, and the model can be run on a conventional Windows platform. Reviewers of the model will be provided with a CD (or DVD) containing a run-time version of the software, one or more executed models, and reference materials. Requirement for Flexibility The actual models are entirely user-defined: the software user constructs their model by inserting and interconnecting objects that represent different components of the model, such as: • Input data values. • Probability distributions for uncertain or stochastic variables. • Formulas or tables defining relationships.

• • •

Waste packages and transport pathways. Receptors. Links to external software modules (as DLL files) that may be required for specific technical calculations. Thus the actual features, events and processes represented by the model are entirely defined by the user— the software simply provides a framework and a ‘palette’ of object classes. Another feature that provides flexibility is the ability to substitute subsystems within a model. For example, an initial, simplified representation of a subsystem can be replaced by a more sophisticated version by simply copying and pasting the new subsystem into the model. Requirement for Extensibility The basic object classes provided by the software are sufficient for representing most performance assessment models, albeit at a somewhat simplified level. For a system such as Yucca Mountain, however, it was necessary for the software to be able to be extended in order to incorporate more sophisticated sub-models of some processes. This capability was provided primarily by allowing dynamic linkage to user-provided software modules. The Yucca Mountain model uses this capability to integrate a number of special-purpose modules into the total system. Some of the key subsystems that are integrated in this way include subsystems for waste package degradation, unsaturated-zone contaminant transport process, and saturated-zone (aquifer) contaminant transport processes. Figure 2 shows the property dialog where a user defines the linkage between an external module and the main simulation model. Requirement for Quality Assurance The development and testing of the software had to be carried out under quality assurance procedures that were consistent with the requirements of standards such as NQA-1 or ISO-9001. An additional requirement for quality assurance is for the software to support quality assurance of the models that are built with it. In the case of the Yucca Mountain Project, this included capabilities to ‘seal’ portions of the model once they were checked by an independent tester, and capabilities to record the nature of changes between different versions of models. Another quality assurance feature of the software is that its inputs and interconnections use a form of strong typing, such that it is impossible to, for example, use an area where a length is required. Also, automatic unit conversion features minimize the possibility of errors due to inconsistent units. Requirement for Discrete and/or Continuous Simulation Many processes at Yucca Mountain are modeled as continuous in time: corrosion of the waste packages, the thermo-chemical evolution within the mountain, and contaminant transport processes in particular fall into this category. However, other processes occur and impact the model at discrete times: climate change, volcanic or seismic disruptions, or human intrusion fall into this category. Thus, the software had to be capable of a combination of discrete and continuous simulation.

Fig. 2. Example of Interface to User-Provided Sub-Model Requirement for Efficiency The Yucca Mountain performance assessment model is highly complex, and represents the evolution of the repository system and its surroundings for periods of up to a million years. Additionally, it uses the Monte Carlo method to handle uncertainty, which requires hundreds to thousands of individual history simulations (realizations). In order to deliver model runs on a timely basis, the software/hardware system involved had to be able to carry out full individual simulations in an hour or less. By networking large numbers of computers together, the performance assessment group at the Yucca Mountain Project can deliver full probabilistic simulations in a matter of hours. A second type of requirement for efficiency arises from the need to develop numbers of alternative or variant models in short order, in order to respond to specific issues or carry out sensitivity studies. The transparent and user-friendly interface that is used allows the modeler to quickly make and test such changes.

REQUIREMENTS FOR PERFORMANCE ASSESSMENT (PA) MODELS As discussed above, there is a parallel set of requirements for the models themselves, as opposed to the software. Requirement for Traceability The Yucca Mountain PA model represents the culmination of a decades-long scientific investigation that has involved thousands of individuals and cost billions of dollars. The resulting data sets, and conceptual and numerical models, will only be accepted by the regulators (and the courts) if their pedigrees are clearly established. This has required the Yucca Mountain Project to develop and implement procedures to document and save all of the information necessary to demonstrate the origin and validity of these components of the model. Using the software transparency features discussed above, the model incorporates information summarizing and referencing the sources of the data and conceptual models of each component, plus hyperlinks directly to copies of the source documents. For example, Figure 3 illustrates the type of documentation that is directly integrated into the model, in the form of notes associated with individual model elements. In this case the element is a formula that is used to calculate the average percolation flux of groundwater across the repository plane.

Fig. 3. Example of Traceability Information Embedded in Model

Requirement for Transparency Using the software’s support capabilities, the models that are being developed are carefully structured and designed to maximize the transparency of the model. Text and graphics are inserted to improve the models’ clarity, a top-down structure of systems and subsystems is employed, and explanatory information is provided wherever appropriate.

Requirement for Credibility The model components are all developed using a multi-layered technical approach intended to ensure the highest possible credibility for the model. All work is carefully planned before execution, and the plans are independently reviewed, as are the work products. All work is carried out under stringent quality assurance procedures. Independent external groups such as the Nuclear Waste Technical Review Board and groups from the International Atomic Energy Agency review the basis for the models as they are developed. The Nuclear Regulatory Commission reviews the work as it progresses, and will ultimately review the entire safety basis for the repository, including the models. One important technique that is used to establish the credibility of simulation models is the comparison of the model’s results to observed historical data. This may be integrated with the process of calibration, where the model’s parameters are adjusted as appropriate to give an improved match to the historical data. Another way to enhance the credibility of a model is to seek natural analogues6, where a natural system that has some features, events or processes in common with the proposed repository system can be simulated. Uranium ore bodies and ancient natural uranium reactors have been used as natural analogues to help build confidence in the modeling techniques proposed for the Yucca Mountain repository. Requirement for Quality Control It is not trivial to ensure that the model that is built is the model that was intended. For example, the Yucca Mountain models consist of tens of thousands of complex components and their interconnections. A simple misspelling of a variable name in an input field could potentially result in incorrect results. The Yucca Mountain models are composed of numerous subsystems. As each subsystem is completed, it is documented internally using the transparency features described above. Then it is independently reviewed against the documents that present the design basis for the subsystem. Upon completion of this review the reviewer applies an electronic seal to the subsystem, which ensures that so long as the seal is unbroken the subsystem has not been altered. The software automatically records which parts of a model have been altered between successive versions. Combined with the information from intact or broken seals, this allows a reviewer to rapidly confirm that a model is what it is supposed to be.

SUMMARY Simulation models are used increasingly as a valuable tool for waste management activities. This paper has discussed requirements for the simulation software tools and the models that are developed using them, in order that the models can be understood and accepted by all of the parties involved. In the charged multi-stakeholder environment often associated with waste management activities, meeting these requirements is essential if a simulation model is to be of assistance.

REFERENCES

Fishwick, Paul A. Simulation Model Design and Execution, Prentice Hall, 1995. Law, Averill and David Kelton. Simulation Modeling and Analysis, 3rd edition, McGraw-Hill, 2000. 3 See review of tools at http://www.informs.org/Resources/Computer_Programs/Commercial_Software/. 4 See review of tools at http://www.idsia.ch/~andrea/simtools.html 5 Sterman, John. Business Dynamics, McGraw-Hill, 2000. 6 Miller, W. et al. Natural Analogue Studies in the Geological Disposal of Radioactive Wastes, Elsevier, 1994. 1 2