methods of expert elicitation into a streamlined method. Fourteen reliability ... incidents at nuclear power plants (e.g., U.S. Nuclear. Regulator Commission, 2000) ...
Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting - 2015
1447
A SURVEY OF EXPERT ELICITATION PRACTICES FOR PROBABILISTIC RISK ASSESSMENT Ronald Laurids Boring Idaho National Laboratory, Idaho Falls, Idaho 83415 Risk and reliability analyses of human errors or hardware failures sometimes need to enlist expert opinion in areas for which adequate human performance or hardware operational data are not available. Experts enlisted in this capacity provide probabilistic estimates of reliability, typically comprised of a measure of central tendency and uncertainty bounds. Formal guidelines for expert elicitation are readily available, but the methods are often time-consuming and costly to implement. This paper reports the first phase of a research effort to combine disparate formal methods of expert elicitation into a streamlined method. Fourteen reliability analysts were interviewed to identify current practices in expert elicitation.
Not subject to U.S. copyright restrictions. DOI 10.1177/1541931215591314
INTRODUCTION As part of the safety oversight process, international nuclear regulators conduct systematic reviews of incidents at nuclear power plants (e.g., U.S. Nuclear Regulator Commission, 2000). Likewise, nuclear utilities also conduct reviews of events that have occurred or conduct prescriptive analysis to determine the risk of events that might occur. These reviews identify and categorize precursors to potential core damage accident sequences. Though rare, an event precursor is an operating event that is an important element of a postulated core-damage accident sequence. Accident sequences of interest to the regulators are those that would have resulted in inadequate core cooling or core damage if additional failures had occurred. As part of such analyses, failure probabilities are determined. For events that are rare or for which there are little operating data, a series of methods has been developed to generate reliable and valid estimates of failure probabilities using expert judgment. For example, just within the United States, NUREG/CR2255 (Stillwell, Seaver, and Schwartz, 1982), NUREG/CR-2743 (Seaver and Stillwell, 1983), and NUREG/CR-3688 (Comer, Seaver, Stillwell, and Gaddy, 1984) outlined methods for using expert judgments to arrive at error probabilities. These approaches explicated paired comparison, ranking and rating, direct numerical estimation, and indirect numerical estimation techniques applied to error estimation, with a particular emphasis on aggregating the estimates from multiple experts. Expert elicitation involves polling subject matter experts to produce a probability of human error or
hardware failure. The analyst who orchestrates the expert elicitation incorporates expert estimates into an overall risk model. When operational data for hardware or performance data for human operators are not available, expert elicitation provides quantitative measures suitable for inclusion in probabilistic risk assessment (PRA) and human reliability analysis (HRA) models. Expert elicitation has proven especially useful within the international nuclear power industry to ensure compliance within regulated operating parameters. These models ensure optimal safety of power plants by identifying potential contributors to risks and ensuring that these contributors are successfully mitigated in the operation of the plant. METHOD Many formal guidelines for such expert elicitation are available (e.g., Ayyub, 2001; Meyer and Booker, 1990; Seaver and Stillwell, 1983). However, the methods espoused in such guidelines have proven timeconsuming and costly to implement (O’Hagan and Oakley, 2004). To assess ways of streamlining the existing guidelines for expert elicitation without compromising the traceability or validity of the results, current reliability analysts within the nuclear power arena were interviewed to obtain lessons learned for expert elicitation. This paper reports the results of those interviews. The purpose of the interviews was to document current practices regarding expert elicitation by risk and reliability analysts in the nuclear power arena. Fourteen analysts who were engaged in the identification and trending of reliability issues for
Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting - 2015
nuclear power plants were interviewed. These analysts are representative of the types of analysts who conduct regulatory oversight and industry reviews of incidents. The following information is extracted from the interviews and represents the processes analysts have undertaken to elicit expert opinion. It must be noted that these lessons learned should not serve as a best practices guide but rather as a snapshot of current practices. The information is presented in qualitative form. Only points identified by a minimum of three analysts are included in the findings. SUMMARY OF FINDINGS From the interviews, there was considerable consistency in issues identified across analysts, including: 1. Expert elicitation comes into play for novel event scenarios, degraded performance, and common cause failure events. 2. Because many aggregation methods are available, analysts implement a variety of methods for resolving estimation disagreements between experts. 3. Subject matter experts polled for expert elicitation do not always produce information that is suitable for input into a risk and reliability analysis. 4. Analysts sometimes have difficulty in identifying appropriate subject matter experts and in securing time for their input. 5. Expert opinion is frequently required for finetuning PRA models. 6. No analysts indicated any direct training on expert elicitation. When Expert Elicitation is Needed Sometimes there are novel event scenarios encompassing unforeseen plant occurrences that are not included in PRA or HRA models and for which probabilistic data are not known. Unexampled events such as severe accidents in the face of natural disasters illustrate such scenarios. When there are no operating or vendor data to support the probabilistic modeling of these scenarios, it is necessary to obtain expert insight on the scenarios to guide modeling efforts and probabilistic determination. Subject matter experts are the primary resource for understanding these novel events.
1448
Expert elicitation is also required anytime there is system or performance degradation that does not exhibit a clear failure. Examples of this include: •
•
•
Degraded performance in which hardware performs unpredictably or in which an individual is likely to exhibit human error; Conditional events, which when left unattended (e.g., no response from operators or automatically actuated systems), would result in an undesired outcome but that would not result in outright system failure; Event recovery, in which an individual circumvents or corrects an error or fault.
For such circumstances, it is often necessary to estimate the likelihood that the hardware failure or human error could have contributed to system failure, even though the system did not fail. A defining characteristic of degraded performance is its unpredictability. If a component has degraded, can it still perform the required function enough to prevent system failure? Similarly, if an individual is impaired, can he or she still perform the task well enough to prevent a potentially unsafe situation from occurring? Often, there are few data sources for determining the probability estimates and uncertainty characteristics. Degraded performance often also requires enhancing the level of detail found in basic logic models of human performance. Experts are enlisted to augment available data sources or to clarify modeling assumptions. A final circumstance requiring expert elicitation is the common cause failure. A common cause failure involves a single failure or error source affecting multiple components or systems. Unlike novel event scenarios like severe accidents, a common cause failure may occur in a well understood domain. Like degraded performance, a common cause failure may not result in outright failure, at least not for all components or systems. The difficulty in accounting for a common cause failure is in determining whether the scenario occurred in isolation. While a given human error may have triggered an isolated component failure, is the system of components also compromised? If the system is compromised, this will significantly increase the probabilistic risk. It is therefore necessary to enlist additional insights in the form of expert opinion to identify which individual components vs. system of components were affected by a common cause failure.
Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting - 2015
Resolving Expert Differences Time, cost, and risk-significance considerations minimize the frequent use of multiple experts, especially for quick-response or near real-time analyses. Most analysts have consulted subject matter experts informally for assistance in finding data sources, but analysts rarely convene an expert panel to resolve an analysis problem. Typically, an analyst begins by framing the problem space and modeling any variations from the standard PRA or HRA model. For those areas in the model where there is uncertainty regarding modeling or probability assignments, the analyst will contact a suitable subject matter expert to gain additional insights. This subject matter expert generates failure probability estimates or produces insights. If the expert's opinion is reasonable and the analyst concurs, the expert elicitation is incorporated into the analysis. If there is disagreement, the analyst may use any additional information provided by the expert to refine the analysis. This process typically entails conferring with another analyst on the team to ensure the validity of the analysis. The analyst may also determine the significance of the problem to determine a further course of action. If the analyst and expert’s estimations diverge but these differences do not significantly impact the outcome of the analysis, the range of estimations is reported as is without further modification. If, however, the differences do result in significant differences in the final analysis, the analyst often enlists another expert as a tiebreaker. Those analysts in the interviews who had used an additional expert as a tiebreaker had employed a variety of methods, including both formal and informal methods, for aggregating the results. Aggregation methods typically consisted of averaging results where possible or weighting the results across experts. Giving more weight to one expert over another occurred when one expert provided a more detailed or credible explanation of the problem under consideration. In all cases, the uncertainty surrounding an analysis is increased to reflect disagreement between experts or between the expert and the analyst. Suitability of Information from the Elicitation Some subject matter experts have sufficient background in statistics and reliability analysis to produce probabilistic estimates that feed directly into PRA or HRA models. However, many experts are not
1449
fluent at producing probabilistic estimates. It is therefore necessary for the analyst to frame the problem in a manner that will produce useful information. The analysts in the interviews identified three main approaches derived from the expert elicitation literature for framing the problem space for experts: •
•
•
One approach calls for the subject matter expert to give contextual or background information that will allow the analyst to translate the problem to probability space. This approach is also useful when determining how to model the problem space. In some cases, the most productive approach may be to elicit Boolean responses from the expert. The expert is polled with yes/no or true/false questions regarding whether or not a component or system or person would fail under certain circumstances. This method requires the analyst to formulate a graduated series of questions that will yield an approximate probability distribution. Another approach is to equip the expert with information to guide his or her probability estimation. This approach usually entails giving experts fixed anchor points through which the expert can make frequency estimations (e.g., Forester et al., 2007).
The analysts who were interviewed did not consistently employ any single approach, and individual analysts alternated between approaches. The consensus among the analysts was that additional guidance was necessary on the most appropriate methods to elicit probabilistic estimates from experts. Identifying Experts Most interviewees contended that there is no list of subject matter experts readily available for the analysts to use within the nuclear power industry, and it is necessary to “ask around” to determine the most appropriate expert. Experts are often self-classified, and their domain of expertise may not always be appropriately matched to the problem. The process of identifying domain experts is particularly challenging for the newer analyst, who has not yet built up an informal index of domain experts. Given the short turnaround time required for some types of analyses, especially in the face of quick response to incidents,
Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting - 2015
this lack of a ready pool of experts can significantly add to the time necessary to complete the analysis. Once experts are identified, there are sometimes issues with the responsiveness of the experts. Calls for expert elicitation may represent a disruption to the expert’s schedule, and the expert may not be able to devote adequate time to the task in a timely manner. In some cases, the extent to which the expert’s time may be reimbursable is also an issue. Where no formal financial agreement exists between the analyst’s group and the expert, the expert elicitation is an “in kind” contribution. As such, the experts may only be able to provide very limited time and effort in support of the analysis. In the interviews, the analysts stressed the importance of “not wasting the expert’s time” by having a clearly defined problem and by keeping to a minimum the number of times any individual expert is contacted. Analysts employ a variety of methods for contacting experts. Telephone calls, face-to-face consultations, and emails were employed with approximately equal frequency. The analysts emphasized the importance of presenting a carefully framed problem to allow the expert to focus on the specific problem at hand. Several analysts highlighted the importance of providing a written record of all interactions with experts, for example, in the form of an email message to the expert summarizing the expert elicitation and its incorporation in the analysis. This process ensures the traceability of the analysis as well as the validity of the analyst’s interpretation of the expert opinion. Expert Elicitation for Model Changes General PRA and HRA power plant models, such as those maintained by the regulator as well as those found at individual power plants, provide sufficient detail to model common hardware and human contributors to reliability. However, in rare cases where the level of detail is not sufficient to model the specific event, it is necessary to modify the PRA or HRA models by adding or deleting basic event nodes. Expert opinion comes into play in terms of the type of modification that is implemented as well as the analyst’s sensitivity to the impact of the changes to the overall model. The changes are typically coordinated with the group that is responsible for the probabilistic models. Changes to the models can be specific to a particular circumstance or can be incorporated as revisions to the models.
1450
Lack of Formal Training While the analysts made use of the formal guidelines that had been written concerning the topic of expert elicitation, no analyst had received training on facilitating expert elicitations. This lack of training made it difficult to downselect methods and to match the best expert elicitation method to particular circumstances. For this reason, it was labor intensive to review methods amid time-sensitive reliability anlayses. There was a strong desire amid analysts to have suitable training to assist in selection of the formal methods as well as to have a streamlined guideline that has been optimized to meet requirements for most expert elicitation scenarios. Analysts tended to adopt overly conservative estimations when expert elicitation is required; more extensive training on expert elicitation methods could serve to create more realistic (rather than conservative) probabilistic estimates for potential accident sequences. CONCLUSIONS The findings from these interviews highlight current practices and shortcomings in expert elicitation for PRA and HRA. Based on an analysis of the interview findings, the following areas for improvement were identified: 1. Specific guidance is needed to help analysts identify subject matter experts as well as evaluate the quality of their elicitation. 2. Appropriate methods should be clarified that are effective for translating expert opinion into probabilistic measures of hardware failure or human error. 3. Expert elicitation takes two forms—both in shaping PRA and HRA model development and in estimating hardware failure or human error. Guidance should be developed to clarify the value and processes of expert consultation for both modeling and probabilistic estimation. 4. Analysts need additional guidance on the appropriate methods for aggregating the results from multiple experts. 5. Training should be made available to assist reliability analysts in effectively using expert elicitation techniques. 6. While current formal expert elicitation guidelines provide a wealth of methods, it is desirable to have guidance on selecting the appropriate method for
Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting - 2015
specific circumstances and to have a general guideline for use in the most common risk analysis scenarios. DISCUSSION Expert elicitation is used across many domains, including PRA and HRA. While the sample of analysts who were interviewed in this paper represents risk and reliability analysts within the nuclear power industry, many of the findings will generalize across domains. Anytime an expert must be consulted, even if the end result is not a probabilistic estimate, there is the need for guidance on: Identifying the appropriate subject matter expert, Translating expert input into the required domain, and • Combining the input from multiple experts. The issues identified in this paper serve as a fruitful starting point for developing such guidance. • •
DISCLAIMER This work of authorship was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government, nor any agency thereof, nor any of their employees makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately-owned rights. Idaho National Laboratory is a multi-program laboratory operated by Battelle Energy Alliance LLC, for the United States Department of Energy under Contract DE-AC07-05ID14517.
1451
REFERENCES Ayyub, B.M. (2001). Elicitation of Expert Opinions for Uncertainty and Risks. Boca Raton, FL: CRC Press. Comer, M.K., Seaver, D.A., Stillwell, W.G., & Gaddy, C.D. (1984). Generating Human Reliability Estimates Using Expert Elicitation, Volume 1. Main Report, NUREG/CR-3688. Washingtion, DC: U.S. Nuclear Regulatory Commission. Forester, J., Kolaczkowski, A., Cooper, S., Bley, D., & Lois, E. (2007). ATHEANA User’s Guide, NUREG1880. Washington, DC: U.S. Nuclear Regulatory Commission. Meyer, M. A., & Booker, J. M. (1990). Eliciting and Analyzing Expert Judgment, A Practical Guide, NUREG/CR-5424. Washington, DC: U.S. Nuclear Regulatory Commission. O’Hagan, A., & Oakley, J.E. (2004). Probability is perfect, but we can’t elicit it perfectly. Reliability Engineering and System Safety, 85, 239-248. Seaver, D.A., & Stillwell, W.G. (1983). Procedures for Using Expert Judgment to Estimate Human Error Probabilities in Nuclear Power Plant Operations, NUREG/CR-2743. Washington, DC: U.S. Nuclear Regulatory Commission. Stillwell, W.G., Seaver, D.A., & Schwartz, J.P. (1982). Expert Estimation of Human Error Probabilities in Nuclear Power Plant Operations: A Review of Probability Assessment and Scaling, NUREG/CR2255. Washington, DC: U.S. Nuclear Regulatory Commission. U.S. Nuclear Regulatory Commission. (2000). Reactor Oversight Process, NUREG-1649, Rev. 3. Washington, DC: U.S. Nuclear Regulatory Commission.
