OECD/CSNI Workshop on Advanced Thermal-Hydraulic and Neutronic Codes: Current and Future Applications, Barcelona, SPAIN, 10-13 April, 2000.
An Automated Code Assessment Program for Determining Systems Code Accuracy Robert F. Kunz, Gerald F. Kasmala, John H. Mahaffy, Christopher J. Murray Applied Research Laboratory, The Pennsylvania State University, University Park, PA, 16804, USA, Tel.: 814-865-2144, Fax: 814-865-8896, e-mail:
[email protected]
Summary An automated code assessment program (ACAP) has been developed to provide quantitative comparisons between nuclear reactor systems (NRS) code results and experimental measurements. The tool provides a suite of metrics for quality of fit to specific data sets, and the means to produce one or more figures of merit (FOM) for a code based on weighted averages of results from the batch execution of a large number of code-experiment and code-code data comparisons. Accordingly, this tool has the potential to significantly streamline the verification and validation (V&V) processes in NRS code development environments which are characterized by rapidly evolving software, many contributing developers and a large and growing body of validation data. This paper summarizes NRS code assessment issues, as well as the analysis capabilities and mechanical deployment of ACAP, with emphasis placed on their relevance to NRS code development environments.
Introduction In recent years, the commercial nuclear reactor industry has focused significant attention on nuclear reactor systems (NRS) code accuracy and uncertainty issues. To date, a large amount of work has been carried out worldwide in this area (see [1-9], for examples). The United States Nuclear Regulatory Commission has sponsored the present authors to: 1) Survey available data conditioning and analysis techniques, focusing on their appropriateness in NRS code accuracy and uncertainty assessment 2) Develop software to deploy recommended techniques 3) Develop coding and interfaces for the software to enable automated assessment on a large number of data sets so as to facilitate code update efforts and modeling revalidation The ACAP software described herein represents the outcome of this code development effort. In [10, 11], details of the data conditioning and analysis technique survey are presented. In [12, 13], an overview of the design of ACAP is provided with several NRS code application examples which illustrate the software’s capabilities, as well as some of the issues that arise in NRS code 1
OECD/CSNI Workshop on Advanced Thermal-Hydraulic and Neutronic Codes: Current and Future Applications, Barcelona, SPAIN, 10-13 April, 2000.
accuracy and uncertainty assessment. The focus of the present paper is on the third (and most recently pursued) effort. Specifically, a spreadsheet based batch capability for executing ACAP has been developed which enables its use in rapidly providing an automated quantitative assessment of the change in the quality of a thermal-hydraulic analysis code from version to version, and provide more specific information on the range of impact of model changes to such a code. In particular, this capability enables the use of ACAP to further automate the configuration control process for the USNRC’s consolidated code, increasing confidence that code modifications do not produce unexpected degradation in the code’s fidelity, and allowing for quantitative tracking of improvements in the code’s capability. In this paper, the several issues associated with systems code accuracy assessment are first summarized, with emphasis of software development environment needs. These issues have led to the design of ACAP, which includes data pre-processing, data analysis and figure-of-merit assembly elements which are also briefly summarized. Next, the functionality of ACAP in both interactive and batch modes is described, with discussion on the appropriate uses of each.
Issues and Needs The issues associated with nuclear reactor systems (NRS) code accuracy and uncertainty assessment are numerous and complex. They include: • scaling of test data, • discretization, • model setup and other “user issues”, • software reliability, • key parameter selection, • the wide variety and complex features of the transient physics, • the inconsistency of measured and computed comparison quantities, • uncertainty in experimental measurements, and • the large and growing available test matrix data base. Though significant progress has been made in addressing most of these assessment issues, reliable and general tools to quantify NRS code accuracy are not available today. An important contribution to meeting this ideal would be a universally available assessment tool for the developers and users of NRS codes to post-process results in a way that would return quantitative correspondence measures of code-code and code-data comparisons. Such a tool would only address some of the uncertainties in real plant analysis. However, it would be part of a process which validates a code with scaled facility data, thereby contributing to uncertainty estimation in full scale plant simulations. Also, as emphasized herein, such a solution accuracy/correspondence quantification tool could be automated to facilitate NRS code development revalidation tasks. The NRC has established qualitative code-experimental comparison measures such as “excellent, reasonable, minimal and insufficient”. These are well defined [9, 14], and allow a group of experts to study a set of results and produce some meaningful statement on the fidelity of a code. The process is useful for major releases on a code, but is time consuming, especially when it must be applied to large test matrices. Therefore, streamlining this man-power intensive and inherently subjective revalidation comparison process might significantly benefit the NRC code consolidation effort. In particular, doing so would allow code upgrades to be rapidly reassessed and would allow for quantitative tracking of improvements in the code’s capability. Consistent with these observations, the goal of the present work has been to initiate a software framework to address several of the NRS code assessment issues summarized above. In
2
OECD/CSNI Workshop on Advanced Thermal-Hydraulic and Neutronic Codes: Current and Future Applications, Barcelona, SPAIN, 10-13 April, 2000.
particular, the ACAP software package has been developed to objectively and quantitatively compare NRS simulations with data. This package was designed to: • draw upon a mathematical tool kit to compare experimental data and one or more NRS code simulations, • return quantitative figures of merit associated with individual and suite comparisons, • accommodate the multiple data types encountered in NRS environments, • incorporate experimental uncertainty in the assessment, • reduce subjectivity of comparisons arising from the “event windowing” process, • accommodate inconsistencies between measured and computed independent variables (i.e. different time steps), • tie into data bases of NRC test data and code results, • provide a framework for automated, tunable weighting of component measures in the construction of overall accuracy figures of merit for a given comparison, and • be deployable in a convenient batch processing mode to allow for revalidation of code versions against a large quantity of test matrix data and NRS code simulations of previous versions. The remainder of this paper summarizes this development effort.
Data Categorization In [10-12], NRS data types are classified into five categories, in order to provide a basis for assessing individual comparison methods. Specifically, scaled NR facilities are instrumented to provide a fairly wide array of data. These include: I) key parameters tables, II) timing of events tables, III) scatter plots of nominally 0-dimensional data, IV) 1-dimensional (in space) steady state data, and V) time record data. The emphasis of the ACAP project is on the latter three. In particular, data conditioning and analysis techniques were assessed and deployed within ACAP for code-code and code-data comparisons of data Types III, IV and V. Type V data in particular provides a significant challenge for several reasons: • the ubiquitous appearance and relevance of these transient data in reactor systems, • the typically long record (often O(105) time steps) nature of these data, complicated significantly by their non-stationarity and diversity in characteristic features, and • the fairly significant differences that often appear between computed and measured time trace data.
Data Analysis Methods In [10, 11], a number of mathematical data analysis methods from the fields of probability and statistics, approximation theory, time-series analysis, as well as adapted methods utilized in atmospheric/geologic sciences, economic forecasting, aerodynamic stability, demographics, digital signal processing and pattern recognition are reviewed for their applicability in the construction of NRS code-data and code-code comparison measures. The goal of that review was to identify issues and techniques to be considered in the development of an automated simulation rating procedure. The findings of that review defined the scope, user interface and a recommended process for using ACAP. A summary of these issues and findings is provided here: 1) Most of the methods considered can be applied to provide useful quantitative measures of accuracy for at least a subset of NRS data Types III, IV and V. 2) Inappropriate use of some methods can yield incorrect results, that is, return figures of merit that are worse for more accurate simulations. This motivates: - Definition of a robust comparison measure or suite of measures as one that reliably returns better figures-of-merit for superior comparisons and worse figures-of-merit for inferior 3
OECD/CSNI Workshop on Advanced Thermal-Hydraulic and Neutronic Codes: Current and Future Applications, Barcelona, SPAIN, 10-13 April, 2000.
comparisons. - That great care be taken in the selection of the suite of analysis tools chosen for each particular comparison. 3) The inherent limitations to stationary data of most available methods render straightforward application to NRS Type V data less than rigorous. Trend removal techniques can be brought to bear to preprocess the data, thereby yielding more robust comparison measures, especially when deployed in concert with time-windowing. 4) For Type V data, techniques that are intrinsically appropriate for non-stationary data analysis can be utilized in the construction of comparison measures. These include best approximation fits and, most promising in the view of the authors, time-frequency techniques. 5) Inconsistency between the computed and measured independent variable range and basis (i.e. different time steps) motivates the incorporation of resampling and range trimming conditioners within ACAP. Such “synchronization” is required for most comparisons. 6) There is a fundamental lack of rigor in applying basic statistical analysis procedures to most NR systems data. This arises due to non-stationarity of the data and the unavailability of a known distribution of error about its mean. This renders the construction of statistical inference measures suspect at best. Basic statistical difference and correlation measures can be deployed to construct useful figures of merit, but uncertainty bounds should not be inappropriately constructed. 7) Experimental uncertainty can be effectively incorporated in code-data accuracy assessment within the framework of the “tool kit” of analysis procedures considered. Experimental uncertainty should be included with the “raw” experimental data in the code reassessment test matrix. 8) The methods vary widely in range/dimensionality of their returned metrics. This complicates the definition of an overall figure-of-merit, and thereby motivated normalization and range limit scaling in constructing component figures of merit. Specifically, each individual comparison measure is redefined to range from 0 to 1. 9) As indicated in conclusion 2 above, great care must be taken in deploying comparison measures. In particular, for each experimental data set, a demonstrably robust assessment strategy must be developed. The present investigators feel that this requirement defines a process whereby expert assessors “calibrate” and document a suite of robust data analyses for each experimental data set in the code reassessment matrix. This assessment configuration will in general include preconditioning strategies, data comparison measures, figure-of-merit weighting assembly factors, and should be included with the “raw” experimental data in the reassessment matrix. Such configured assessments will then be used to define ACAP sessions in future code re-assessments. In concert with the above design criteria and method assessment findings, a set of baseline techniques for code-data (or code-code) comparisons, data preconditioning, figure-ofmerit-assembly and incorporation of experimental uncertainty were selected and implemented in ACAP. These are summarized in the next section. For brevity, the details of the mathematics associated with these methods are not provided in this paper - the reader is referred to [15].
4
OECD/CSNI Workshop on Advanced Thermal-Hydraulic and Neutronic Codes: Current and Future Applications, Barcelona, SPAIN, 10-13 April, 2000.
ACAP Methods The data conditioning and data comparison utilities available in ACAP are summarized in Table I. The available data conditioning utilities include particular choices of resampling, trend removal and time windowing methods. A number of authors have observed the usefulness of time-windowing of transient NRS data in order to isolate distinct physical processes, and thereby provide a more focused assessment of simulation strengths and weaknesses. Resampling of the computed data traces is usually appropriate in order that the experiment and simulation have a consistent independent variable basis (i.e. discretization increments relevant to Types IV and V data). Lastly, the authors have found that trend removal techniques can be useful in analyzing non-stationary NRS data. In particular, a trend removal step allows the independent assessment of the accuracy with which large scale trends and higher frequency oscillatory features are predicted, resulting in a more robust comparison configuration.
Table I. ACAP Methods
5
Method
Utility Class
D’Auria FFT (DFFT)
Data Comparison
Mean Error (ME)
Data Comparison
Variance of Error (VE)
Data Comparison
Mean Square Error (MSE)
Data Comparison
Mean Error Magnitude (MEM)
Data Comparison
Mean Relative Error (MRE)
Data Comparison
Index of Agreement (IA)
Data Comparison
Systematic Mean Square Error (SMSE)
Data Comparison
Unsystematic Mean Square Error
Data Comparison
Mean Fractional Error (MFE)
Data Comparison
Cross-Correlation Coefficient (ρxy)
Data Comparison
Standard Linear Regression (L2-standard)
Data Comparison
Origin Constrained Linear Regression
Data Comparison
Perfect Agreement Norm (L2-perfect
Data Comparison
Continuous Wavelet Transform (CWT)
Data Comparison
Percent Validated (PV)
Data Comparison
Resampling
Data Conditioning
Trend Removal
Data Conditioning
Time-Windowing
Data Conditioning
Among the data comparison utilities, the reader will likely recognize the FFT method of D’Auria [5]. Also available are a number of baseline statistical techniques (methods 2-5, 11-14), Willmott’s Index of Agreement (method 7, [16]) and several adapted statistical methods utilized by the atmospheric sciences community (methods 6, 8-10, see [17] for example). Experimental uncertainty is incorporated in a fashion consistent with recent computational fluid dynamic (CFD) code validation work undertaken by Coleman and Stern [18], where a “Percent Validated” metric (method 16) is defined from the fraction of simulation data in a trace which falls within the uncertainty bands of the measurements. A particularly attractive comparison tool for NRS code accuracy assessment is the continuous wavelet transform (CWT) measure installed in ACAP (method 15), due to the direct applicability of wavelet analysis to nonstationary data. Significantly more detail on the specific methods employed in ACAP are available in [10-13, 15]. Also, in these references, several examples of application of the code to sample nuclear reactor systems code test cases
OECD/CSNI Workshop on Advanced Thermal-Hydraulic and Neutronic Codes: Current and Future Applications, Barcelona, SPAIN, 10-13 April, 2000.
are included which illustrate the capabilities of the tool. There, the software’s ability to provide objective accuracy assessment figures are demonstrated. It should be re-emphasized that though it is demonstrated in the cited references that the methods which define the underpinnings of ACAP can be deployed usefully in code assessment, great care must be taken in using them in a demonstrably robust fashion.
ACAP Program Description and Mechanics ACAP is a PC and UNIX station based application which can be run interactively on PCs running WINDOWS 95/98/NT or in batch mode on PCs as a WINDOWS console application or in batch mode on UNIX stations as a command line executable. The code was delivered to NRC with full source code. The interactive and batch PC versions can be modified and recompiled from a WINDOWS “folder” under the Microsoft Visual C++ environment. The batch UNIX version can be modified and recompiled using any C++ compiler which conforms to the C++ draft standard (including the freely available g++/gcc compilers). Interactive Mode Execution A brief summary of the operation of ACAP is provided here. Figure 1 shows a schematic overview of the structure of the code. Experimental and computational NRS data are input through ACAP data files, which, in their simplest form, contain a table of x-y data and a few data descriptor keywords. The user specifies, either interactively or through front end script files, a suite of data conditioning and data analysis methods to be deployed in quantifying the correspondence between the measurements (if available) and the (one-or-more) simulation data sets. This suite of methods is termed the ACAP configuration, which can be saved in a file for later use on the current or other data sets. In interactive mode, ACAP displays the data sets with a modest but reasonably versatile embedded plotting package, and provides standard windows environment interfaces to select and adapt the mathematical methods to be deployed. The code then executes specified data conditioning processes and data comparison measures. Lastly, with user selected weighting, also part of the configuration, an overall figure-of-merit is constructed quantifying the accuracy of the individual code runs. The results of the ACAP session, including a summary of all selections made, and the component and overall figures-of-merit are output to screen and file. Figure 2 illustrates several elements of the interactive ACAP interface for an application of the software to the "D’Auria" data [5]. Complete documentation for the stand-alone ACAP software is available in the ACAP User’s Manual (Appendix A of [13]). Batch Mode Execution As discussed above, users must take care in assembling robust ACAP configurations, in order that returned FOMs reliably quantify the improvements or degradation in model upgrade/ code version. The interactive mode for ACAP is preferred in constructing these configurations on a "new" test matrix data set. This is because the ACAP GUI allows one to effectively visualize and interact with the systems code and experimental data (i.e. try different conditioning and comparison strategies) until a satisfactory configuration is established. Once a configuration is established for a given test suite entry, it becomes, in principle, frozen in time. Subsequent reassessments of code versions are then more efficiently carried out in batch mode. ACAP has the potential to significantly streamline NRS code development efforts. The development environment for such software is characterized by rapidly evolving software (i.e.
6
OECD/CSNI Workshop on Advanced Thermal-Hydraulic and Neutronic Codes: Current and Future Applications, Barcelona, SPAIN, 10-13 April, 2000.
frequent updates), many contributing developers and a large (and growing) body of validation data. As each new version of a NRS code is proposed for release, it is important that a revalidation process is undertaken, on some level, to ensure that new modifications have not “broken” some of the required application capabilities of the code. Such revalidation is a major element in the configuration control of the USNRC's consolidated code, and allows for quantitative tracking of improvments in the code's capability. ACAP, running in batch mode, has great potential to further expand this role in the development process. Accordingly, batch execution of ACAP is provided as an option within the auto-validation tool (Auto-DA), currently in use at NRC. As also illustrated schematically in Figure 1, this tool automatically runs systems code simulations for a sequence of test cases and generates a prespecified series of plots, using xmgr5 [19], which include experimental measurements and the results of the multiple simulation runs. The Auto-DA utility, which is comprised of two PERL scripts, has been extended to optionally execute ACAP for each test case, so that attendant to each test case and plot is the figure-of-merit output of the ACAP session.
AUTO-DA ACAP
Spreadsheet specification of testcases, xmgr5 plotting parameters and ACAP configurations Interactive data display
Data conditioning
Conversion to text files
Successively execute NRS codes for specified cases/code versions
Generate xmgr5 batch execution file
Data importing
Synchronization, trend removal, time-windowing utilities
Select analyses and FOM weighting/ assembly, error checking
Data comparison utilities
Perform data analyses
FOM weighting/assembly Execute xmgr5, generate postscript plots
Generate overall FOM/log file
Generate ACAP data, script and execution files
Figure 1. Schematic overview of the structure of ACAP and the Auto-DA tool. This Auto-DA/ACAP batch capability is illustrated here with an example application. Elements of this example are provided in Figure 3. Figure 3a shows the Auto-DA "path" spreadsheet page (Microsoft Excel shown here), which points to two TRAC-M executable version file paths and the ACAP executable file path. The Auto-DA "cases" spreadsheet page is shown in Figure 3b. There, the "CaseIDs" are identified, Demo1, Demo2, Demo3. Two rows and associated TRAC-M version are included for each case, indicating that both the baseline and a "new" version of TRAC-M are to be executed for this case. The "Base" column indicates with an "X" which run is 7
OECD/CSNI Workshop on Advanced Thermal-Hydraulic and Neutronic Codes: Current and Future Applications, Barcelona, SPAIN, 10-13 April, 2000.
a)
b)
c)
Figure 2. Elements of interactive mode ACAP interface. a) "D’Auria" data [5] displayed in ACAP main window with results of comparison assessment for sample “code” results. b) Resampling dialog. c) Figure-ofmerit configuration dialog.
8
OECD/CSNI Workshop on Advanced Thermal-Hydraulic and Neutronic Codes: Current and Future Applications, Barcelona, SPAIN, 10-13 April, 2000.
to be deemed the base case against which all other runs for that case (only one here) are to be compared using ACAP. (Often an experimental data set will be the "base" and one or more NRS code runs will be compared to it in ACAP). The "ACAP" spreadsheet page illustrated in Figure 3c includes, for each of the cases, a sample fully configured ACAP session. Specifically, all information necessary to run ACAP once for each case (i.e., three times here [Demo1, Demo2, Demo3]) is provided including data conditioning, data comparison and FOM assembly elements. Once these spreadsheet pages are assembled, they are converted to text files and Auto-DA is executed. For those cases that an ACAP configuration has been built, Auto-DA generates the two necessary input files to ACAP. One is a data file which contains x-y values for the data sets to be assessed. The other is an ACAP script file which names the data file and specifies the ACAP configuration (processing parameters such as preprocessing to be performed, metric selection and parameters as required, and metric weighting). The Auto-DA script generates these two files for the specified NRS code solutions and experimental data as well as a file which causes ACAP to the launched with the appropriate ACAP script for execution external to the Auto-DA script. ACAP then executes once for each case, writing computed component and overall figuresof-merit to a file. Once the input spreadsheets are assembled for a given suite of test cases, the overall process outlined above becomes quite streamlined. In particular, to revalidate a new version of the code one simply edits the "path" and "cases" pages to point to the new code version and reexectues Auto-DA and its output ACAP script.
ACAP Configurations In the above example an arbitrary ACAP configuration was specified which executes all available ACAP methods (Figure 3c). As discussed above, robust ACAP configurations must be assembled for each test in a revalidation suite. For complex data sets (i.e. those with rich transient features such as long time scale damping, local quasi-periodicity, sudden changes due to active or passive phenomena, chatter (often of high amplitude), dependent variable limits [for volume fraction] between 0 and 1) this can be difficult, indeed definitive procedures to do so remain unavailable. However, for revalidation applications, where NRS code solutions do not vary dramatically, the authors provide here "lower fidelity” recommendations for building configured ACAP sessions for Types III and V data. These baseline configurations are relatively simple but capture several of the basic requirements/features of an optimum configuration. Accordingly they are probably useful on some level, and also probably represent a good starting point for ACAP users who wish to become familiar with the code and/or a new Type III or V data set. However, these recommendations do not necessarily form the basis for robust configurations of the fidelity required for test matrix reassessment. • Type III Data Configurations: Data conditioning configuration: N/A Figures-of-merit configuration: Methods - Index of Agreement, IA - Mean Fractional Error, MFE - Cross-correlation coefficient, ρxy Normalization: - Equal weighting for all selected metrics Discussion:
9
OECD/CSNI Workshop on Advanced Thermal-Hydraulic and Neutronic Codes: Current and Future Applications, Barcelona, SPAIN, 10-13 April, 2000.
a)
b)
c)
Figure 3. Elements of batch mode Auto-DA/ACAP spreadsheet interface. a) Auto-DA "path" spreadsheet page, b) "cases" page, c) "ACAP" page
10
OECD/CSNI Workshop on Advanced Thermal-Hydraulic and Neutronic Codes: Current and Future Applications, Barcelona, SPAIN, 10-13 April, 2000.
MFE and ρxy metrics capture the distinct features of bias and correlation of the data, respectively. Additionally, the basic statistical IA metric has been widely used for Type III data in the Atmospheric Sciences community, where the data are not rendered “0-D” by collapsing data obtained at multiple space-time coordinates to a single scatter plot. • Type V Data Configuration Data conditioning configuration: - Resample using linear interpolation to experimental time steps. Figures-of-merit configuration: Methods: - D’Auria FFT - Mean Error or L2-standard - Percent Validated, PV (if experimental uncertainty is available) Continuous Wavelet Transform, CWT Normalization: - Equal weighting for all selected metrics Discussion: The mean error and L2 norm are the most straightforward measure of absolute error. Additionally, the D’Auria FFT metric has been fairly widely used for Type V data in the NRS code assessment community. As discussed above and in detail in [10-12], the CWT metric has the potential to capture a wide variety of distinct features of the error between data and simulation. The appropriateness of these basic Type III and V configurations is illustrated with two example applications in [12].
Conclusions The Automated Code Assessment Program (ACAP) software, developed by the authors under contract to the USNRC, has been presented. Several issues related to systems code accuracy assessment were summarized as these play importantly in the design of ACAP. An overview of the code mechanics with example illustrations was provided with emphasis on the recently implemented batch mode Auto-DA interface which can play an important role in configuration control role in the evolution of the NRC’s consolidated code.
References [1] Bessette, D.E., Odar, F., “The U.S. Nuclear Regulatory Commission (NRC) Program on the Development and Assessment of Thermal Hydraulic Systems Codes,” NUREG/CP-0080, Vol. 1 (1986). [2] Kmetyk, L.N., Byers, R.K., Elrick, M.G., Buxton, L.D., “Methodology for Code Accuracy Quantification,” NUREG/CP-0072, Vol. 5 (1985). [3] Wilson, G.E., Case, G.S., Burtt, J.D., Einerson, J.J., Hanson, R.G., “Development and Application of Methods to Characterize Code Uncertainty,” NUREG/CP-0072, Vol. 5 (1985). [4] D’Auria, F., Leonardi, M., Glaeser, H., Pochard, R. “Current Status of Methodologies Evaluating the Uncertainty in the Prediction of Thermal-Hydraulic Phenomena in Nuclear Reactors,” from Two-Phase Flow Modeling and Experimentation, 1995, ed: Celata, Shah, pp. 501-509 (1995). 11
OECD/CSNI Workshop on Advanced Thermal-Hydraulic and Neutronic Codes: Current and Future Applications, Barcelona, SPAIN, 10-13 April, 2000.
[5] Ambrosini, W., Bovalini, R., D’Auria, F., “Evaluation of Accuracy of Thermal-Hydraulics Code Calculation,” Energianucleare, Vol. 7, No. 2, pp. 5-16 (1990). [6] D’Auria, F., Galassi, G.M., “Code Assessment Methodology and Results,” IAEA Technical Committee/Workshop on Computer Aided Analysis, Moscow (1990). [7] D’Auria, F., Debrecin, N., Galassi, G.M., “Outline of the Uncertainty Methodology Based on Accuracy Extrapolation,” Nuclear Technology, Vol. 109, January, pp. 21-38 (1995). [8] D’Auria, F., Galassi, G.M.“Code Validation and Uncertainties in System Thermalhydraulics,” submitted Journal of Progress in Nuclear Energy (1997). [9] Schultz, R.R., “International Code Assessment and Applications Program: Summary of Code Assessment Studies Concerning RELAP5/MOD2, RELAP5/MOD3, and TRAC-B,” NUREG/IA-0128 (1993). [10] Kunz, R.F., Mahaffy, J.M. “A Review of Data Analysis Techniques for Application in Automated Quantitative Accuracy Assessments,” International Meeting on "Best-Estimate" Methods in Nuclear Installation Safety Analysis (BE-2000), Washington, DC, November, 2000. [11] Kunz, R.F., Mahaffy, J.M. “Task Order #3, Letter Report 1, Literature Review, Description and Demonstration of Techniques,” USNRC Contractor Report, January, 1998. [12] Kunz, R.F., Kasmala, G.F., Murray, C.J., Mahaffy, J.M. "Application of Data Analysis Techniques to Nuclear Reactor Systems Code Accuracy Assessment", Presented at the IAEA Conference on Experimental Tests and Qualification of Analytical Methods to Address Thermalhydraulic Phenomena in Advanced Water Cooled Reactors, Villigen, Switzerland, September, 1998. [13] Kunz, R.F., Kasmala, J.K., Mahaffy, J.M. “Task Order #3, Completion Letter Report,” USNRC Contractor Report, November, 1998. [14] Damerell, P.S., Simons, J.W. [editors] (1993) “2D/3D Program Work Summary Report,” NUREG/IA-0126. [15] Kunz, R.F., Kasmala, J.K., Mahaffy, J.M. “Task Order #3, Letter Report 3, Automated Code Assessment Program: Technique Selection and Mathematical Prescription,” USNRC Contractor Report, April, 1998. [16] Willmott, C.J., “Some Comments on the Evaluation of Model Performance,” Bulletin of the American Meteorological Society, Vol. 63, No. 11, pp. 1309-1313 (1982). [17] Ku, J.Y., Rao, S.T., Rao, K.S., “Numerical Simulation of Air Pollution in Urban Areas: Model Performance,” Atmospheric Environment, Vol. 21, No. 1, pp. 213-232 (1987). [18] Coleman, H.W., Stern, F., “Uncertainties and CFD Code Validation,” Journal of Fluids Engineering, Vol. 119, No. 4 (1997). [19] http://www.nrc.gov/RES/RELAP5/xmgr.html
12
