ELICITING PUBLIC VALUES FOR COMPLEX POLICY DECISIONS *

MANAGEMENT SCIENCE Vol. 36. No. 9, September 1990 Prhred it1 U.S.A.

ELICITING PUBLIC VALUES FOR COMPLEX POLICY DECISIONS * RALPH L. KEENEY, DETLOF VON WINTERFELDT A N D THOMAS EPPEL Institute ofsafety and Systems Management, University of Southern California, Los Angeles , California 90089 Krannert Graduate School of Management, Purdue University, West Lafayette, Indiana 47907 Several approaches exist to illuminate and clarify public values relevant for making public policy decisions. These include surveys, indirect and direct value elicitation, focus groups and public involvement. This paper describes a new approach, called the public value forum, which combines elements of focus groups and direct multiattribute value elicitation techniques. Two public value forums were conducted with selected members of the West German public to elicit values relevant for setting long term energy policies. The purposes of conducting the value forums were to examine the feasibility of eliciting values from laypeople and combining them with factual assessments of experts, to determine the extent to which values elicited formally conflict with values elicited informally, and to assess the advantages and disadvantages of the public value forum. The results indicate that the public value forum is feasible, that the participants felt comfortable with the procedure and that they were eager to resolve inconsistencies between their intuitive judgments and the multiattribute models. There was substantial conflict between the formally and informally elicited values. However, the participants were able to resolve those conflicts in the course of the value forum, tending towards more moderate alternatives in the process. The public value forum provided useful information for the policy process and education for the participants. However, because it is expensive and time consuming, its main application may involve small samples of opinion leaders and stakeholder representatives, rather than large representative samples of the general public. (PUBLIC VALUES; POLICY DECISIONS; DECISION ANALYSIS; MULTIATTRIBUTE UTILITY ANALYSIS)

I . Incorporating Public Values in Policy Decision Making Voting for political representatives is the main mechanism for incorporating public values in policy decision making in a democratic society. Since preferences for candidates reflect preferences in values, one assumes that these candidates will attempt to evaluate policy decisions in a way that is consistent with the values of their constituency. While voting and political representation is and will remain the main vehicle for incorporating public values, it leaves unresolved how political representatives or policy makers should interpret public values in a specific policy context, how public values should be operationalized, what role the experts and their values should have, and how expert recommendations and value interpretations should be combined in policy making. These issues become more complex as the policy context moves from local to national and from fairly simple issues like land use and waste disposal to highly complicated issues like choices of energy strategies or of national security policies. Several approaches can help to illuminate and clarify public values in complex policy problems: surveys, indirect and direct value elicitation, focus groups and public involvement. Each of these alternatives has its advantages and disadvantages. Szuveys can obtain information about public values in the form of priorities among objectives (e.g., what do you think is more important: the environment or employment?), opinions about alternatives (e.g., how risky do you think nuclear energy production is?), and preferences among alternatives (e.g., do you prefer nuclear energy production over

* Accepted by Robert L. Winkler, former Departmental Editor; received December 15, 1988. This paper has been with the authors 3; months for I revision. 101 1 0025- l909/90/3609/ 101 1$01.25 Copyright O 1990, The Institute of Management Sciences

10 12

RALPH L. KEENEY, DETLOF VON WINTERFELDT A N D THOMAS EPPEL

coal energy production?). Surveys merely inform the policy process and are not binding for policy making. Sophisticated surveys examine the detailed structure of respondents' attitudes about policy alternatives, including separate judgments about factual opinions and values (see, e.g., Fishbein and Ajzen 1975). Among the disadvantages are the hypothetical nature of the questions, the influence of the survey designer (framing and response mode effects), and the usual difficulties in designing, administering and interpreting surveys. Indirect Elicitation of Public Values. Instead of asking people directly, one can also infer public values from their behavior in the marketplace (see, e.g., Mishan 1976; Merkhofer 1987). In particular, prices for various commodities and previous actions involving explicit or implicit value tradeoffs can provide some guidance about public tradeoffs among objectives. For example, if consumers are willing to spend $500 for a safety device on a car, this willingness to pay indicates the value of safety to them. The main problem with this indirect approach to inferring public values is that many values do not have a simple observable economic equivalent. Economists have therefore developed the "contingent valuation" techniques, in which samples of the public are asked for their willingness to pay for, say, environmental quality or health improvements, in hypothetical market situations (see, e.g., Merkhofer 1987; Cameron and James 1987) . The advantages of the contingent valuation approach are that it can be applied to nonmarket items as well as to market items and that it provides the policy maker with a fairly direct set of guidelines for the tradeoffs that people consider reasonable. The main problems with the contingent valuation approach are that the questions are hypothetical, that in many cases the hypothetical market problems mix value a ~ factual d information and that subtle changes of the way the questions are asked can lead to important changes in the answers. Direct value elicitation consists of interacting with individuals or groups to elicit the value dimensions on which they would like to see the policy options to be evaluated and the tradeoffs among units of these dimensions. The elicitation involves only questions about preferences and tradeoffs and not questions about the factual side of the policy alternatives. There are numerous procedures for formal value elicitation ranging from approximate rating and weighting methods (see, e.g., Edwards 1977) to tradeoff methods in the framework of multiattribute utility analysis (see, e.g., Keeney and Raiffa 1976). Like contingent valuation methods, these methods can contribute to improve the policy process by providing the tradeoff information that policy makers need when making tough decisions (e.g., how much should society pay to increase the safety of a facility?). Unlike contingent valuation methods, they carefully separate out the value side from the factual side of the policy problem. Disadvantages include the cost and time to elicit the information, the hypothetical nature of the tradeoff questions and the difficulties that laypeople may have in answering them. Focus groups have been developed in the marketing and advertising area in order to learn about people's values, needs, concerns, and preferences about products. A focus group typically meets for several hours to one day and involves five to ten participants. The format is fairly open and the usual task is to test out ideas and concepts on a group that is part of the eventual target population of the marketing or advertising effort. The use of focus groups in public policy problems is relatively new (see, e.g., Desvogues and Smith 1988). The advantages of focus groups are that much value relevant information can be exchanged and gathered, that the focus group leaders can be truly adaptive to participants' needs and that the format is very flzxible. The disadvantages are that the groups are small and unrepresentative, that the information collected may be merely anecdotal, and that there are no specific guidelines on how to elicit value relevant information that can be useful for policy makers. Public involvement is a form of direct interaction with members of the public (see, e.g., Creighton et al. 1982) that is intended to solve a specific problem. It typically consists

ELICITING PUBLIC VALUES FOR COMPLEX POLICY DECISIONS

1013

of a series of meetings and hearings involving experts and policy makers on one hand and selected members of the public and interest groups on the other. The type of interaction ranges from purely informational to a joint problem solving process. Public involvement shares many advantages and disadvantages of focus groups. It has a distinct advantage over focus groups in that it is intended to solve a specific problem that is of concern to the participants, as opposed to simply testing reactions to proposed solutions. Its disadvantages include the necessary restriction to fairly small problems that can be resolved with public involvement and the possible constraints on the decision making of the agency by the involvement activity. In this paper, we will investigate a combination of focus group and direct value elicitation techniques. In particular, we will examine the feasibility and usefulness of using a multiattribute utility based tradeoff procedure to elicit value relevant information from a focus group and to arrive at preferences for policy alternatives by combining this value information with expert assessments of the policy alternatives. The approach builds on previous attempts to elicit public values for policy decisions concerning water resources (Brown 1984), energy systems (Renn et al. 1984), and offshore oil development (von Winterfeldt 1987). The specific combination of focus groups and direct value elicitation used in this study is called the "public value forum." In brief, the public value forum consists of a workshop or a series of workshops in which selected members of the public produce a catalogue of objectives that they consider relevant for evaluating policy alternatives. These objectives are operationalized using attributes and the participants of the value forum are asked to express tradeoffs among the attributes. Their tradeoffs are combined with expert assessments of the performance of the policy alternatives on the attributes to generate a multiattribute evaluation of the alternatives. The main results of this process are the list of objectives and attributes that the participants consider value relevant, the tradeoffs that they consider reasonable, and the preferences that constitute their recommendation to policy making. This information can be used in the policy process to guide further analyses, to narrow down tradeoffs for sensitivity analysis and to generate new options. The public value forum was applied in an experimental setting in the context of evaluating complex energy policies in the Federal Republic of Germany (FRG). Building on previous work by the German parliament's "Enquete Commission Future Energy Policy," the task was to evaluate four alternative energy scenarios, that ranged from a strict conservation and slow growth scenario with elimination of the use of nuclear power to a relatively high growth scenario with reliance on coal, conventional nuclear power, and the fast breeder. Within this problem context, the present study attempted to elicit public values about the four energy scenarios in two two-day meetings with selected members of the German public. The purposes of the study were ( a ) to examine the feasibility of the public value forum, especial1.y whether values can be systematically elicited from nonexperts and combined with the factual inputs from experts; ( b ) to assess the extent to which values elicited formally conflict with values elicited informally, and whether these discrepancies can be resolved by laypeople; ( c ) to determine, whether values elicited without specific knowledge of the policy alternatives conflict with values elicited with that knowledge, and whether such discrepancies can be resolved by laypeople; ( d ) to assess the advantages and disadvantages of the value forum as an input to the policy debate. The paper is organized as follows. In $2 we will describe the basic process and structure of a public value forum. In $3 we will describe the procedure of two public value forums in which a sample of 23 German citizens provided value inputs to the energy policy debate in West Germany. The results of this value forum will be described in $4 and $5

1014

RALPH L. KEENEY, DETLOF VON WINTERFELDT AND THOMAS EPPEL

concludes with a discussion of the advantages and shortcomings of the public value forum. 2. The Process and Structure of a Public Value Forum The public value forum is a meeting of members of the public or interest groups that typically lasts one or two days and involves between five and 25 participants. Because the time of the participants and the forum staff is valuable, it is important to carry out several structuring activities prior to the forum. Most importantly, the policy problem has to be formulated as a concrete decision problem by defining specific policy alternatives and by constructing a value tree of objectives and attributes on which the alternatives should be evaluated prior to the forum. In some cases, it is useful to have a series of separate meetings with representatives of stakeholder groups involved in the policy problem in order to define alternatives, objectives, and attributes. Next, members of the public have to be identified and selected for participation in the forum. There are two basic approaches to selecting participants: the stakeholder approach and the representative approach. In the stakeholder approach, groups that have a strong interest in the policy problem and have clear values and preferences are identified for participation and value forums are carried out with each group. The stakeholder approach is most useful, when the policy issue is highly controversial, when the emphasis is on negotiation and conflict resolution, and when there is hope of achieving some kind of negotiated compromise. In the representative approach, members of the public are selected at random to participate in one or a series of value forums. The representative approach is most useful, when there exists very little knowledge about reasonable public values, when the emphasis is on broad policy issues rather than on resolution of specific conflicts, and when accountability of the value judgments is important. The general structure of the actual meeting consists of the following steps: 1. Introduce the policy problem and motivate the value forum; 2. Refine the objectives and attributes; 3. Elicit single-attribute utility functions from the participants; 4. Elicit tradeoffs among the attributes from the participants; 5. Combine the results of 3 and 4 with expert judgments to evaluate policy alternatives; 6. Reconcile participants' intuitive and modelled evaluations of the alternative policies. In the following, we will describe these steps in more detail. 1. Introduction and Motivation. In the early stages of the forum, care should be taken to explain the policy alternatives to the participants, to clarify the distinction between factual inputs by the experts and value inputs needed to make decisions and the importance of using public value judgments in the decision making process. 2. Refinement of Objectives and Attributes. The value tree constructed prior to the forum should be presented to the participants and discussed. Care should be taken that the objectives and attributes cover all the concerns of the participants and that their meanings are clear. Modifications to the value tree structure, the objectives, and the attributes should be made as indicated by the discussion. 3. Elicitation of Single-Attribute Utility Functions. It is not always the case that the quantitative levels of the attributes elicited in step 2 are appropriate reflections of their relative desirability or utility. For example, if national energy independence is measured in percent of primary energy imported (vs. total), there may be a steep decline in utility between 10 and 50%, after which the percentage almost does not matter, since things "could hardly get worse." Methods for constructing utility functions range from simple rating methods to fairly complex indifference methods (see Keeney and Raiffa 1976; von Winterfeldt and Edwards 1986). The choice of method depends on the purpose of the value forum. In many instances simple rating methods are probably sufficient. If the


1015

shape of the utility function is very important, more precise bisection or indifference methods might be preferable. Because utility functions reflect personal preferences, singleattribute utility functions should be elicited from each participant individually. 4. Elicitation of Tradeofs. Tradeoffs among attributes express the relative importance of attribute units by defining the exchange rate of one attribute unit vs. another. For example, some people may think that both worker's health risk and public health risks should be valued equally in policy decisions and that a statistical fatality should be valued at $1,000,000. There are many methods for eliciting tradeoffs and relative importance information. The methods range from approximation procedures like ratio or swing weighting (see von Winterfeldt and Edwards 1986) to indifference procedures and pricing out methods (see Keeney and Raiffa 1976). The choice of the appropriate method depends again on the policy context and the purpose of the value forum. If the value forum is to provide specific guidance about tradeoffs in a formal evaluation of policy alternatives, the more specific and precise indifference procedures should be used. If, on the other hand, the need is only for approximate information about relative importance, ratio weighting and swing weighting may be adequate. Because tradeoffs reflect personal preferences, they should be elicited individually. 5. Construction of a Mtiltiattribute Utility Model and Evaluation of Alternatives. The tradeoff information elicited in step 4 is converted into weights for the attributes using standard multiattribute utility techniques (see Keeney and Raiffa 1976). In most cases, a multiattribute utility model is a simple weighted average of the single-attribute utilities. Tests can be performed to examine more complex multiplicative or multilinear models (see Keeney and Raiffa 1976), if the additive model is questionable. If more complex models are chosen, additional tradeoff questions may need to be asked to elicit the additional parameters of the models. The multiattribute utility model is used in combination with the expert assessments of the single-attribute performance of the policy alternatives to generate an overall evaluation. In the additive case, the overall utility u for an alternative described by x is

where n is the number of attributes, Wi is the weight for the ith attribute, ui is the ith utility function and xi is the expert assessment of the impact of the alternative described by x on attribute i. When xi is not known precisely, it is often appropriate to represent the uncertainty about it by a probability distribution Zi constructed from expert judgment and to substitute the expected utility of Zi for ui(xi). When experts strongly disagree about the xi's, it is preferable to use several xi's and carry out sensitivity analyses with them to determine whether they change the overall rank order of the alternatives. In most cases, the wi's and the u i's are personal, and the overall utility models should therefore be calculated for each value forum participant separately. Often, it is useful to calculate certain statistics, like average weights and average utilities, to compare groups of individuals. This is particularly important in cases that involve multiple stakeholders. 6. Discussion and Resolution of Uncertainties. One of the most interesting parts of any multiattribute utility elicitation process, and certainly an important part in the public value forum, is the reconciliation of the disagreements between the calculated results and the intuitive preferences of the participants. In most cases this should be done on an individual level. Participants should be told how the model was calculated, and that it is perfectly normal if the model results disagree with their intuitive preferences. Resolution strategies should be explained and assistance should be offered in reconciling the results. The results of a public value forum are sets of compact descriptions of value models that represent participants' value inputs into the specific policy debate. They can be used,

10 I6


like surveys or results of a focus group, as informal inputs into the policy discussion. They can also be used more formally as guidelines for the tradeoffs that are to be applied to the issue at hand, or as a mechanism for analyzing and resolving conflicts among those that strongly disagree in their values.

3. The Public Value Forum for Evaluating West Germany's Energy Scenarios The public value forum was experimentally applied in two meetings with West German citizens as part of an effort to assist the West German parliament in evaluating alternative energy policies for the next 50 years (see also Renn et al. 1984). First, the energy problem was structured in terms of four long-term energy "paths" that could lead West Germany to a sustainable energy future. These paths were developed by the "Enquete Commission on Energy Policy" which had been set up by the German parliament (Enquete Commission 1980). The four paths were primarily described by the technical characteristics of the energy supply options and the changes in the supply mix over the next 50 years. TABLE 1 Characteristics of Four "Energy Paths" Developed by the Enquete Commission

Path 1 continuing high import of oil and gas increased import of coal and uranium use of substantial amounts of fossil fuels substantial increase of nuclear energy (use of fast breeders and recycling of burnt material) substantial increase of the number of power plants

Trend generous use of energy Path 2 decreased import of oil and gas increased import of coal increased use of domestic coal increase of nuclear energy support for new technologies

Trend efficient use of energy Path 3 decreased import of oil and gas increased import of coal increased use of domestic coal stepwise reduction of nuclear energy use of regenerative energy sources strong support for new technologies

Trend limiting the use of energy Path 4 minimal use of oil and gas increased use of domestic coal no use of nuclear energy maximum use of regenerative energy sources substantial use of new technologies

Trend Extreme saving of energy and strong use of regenerative energy sources

ELICITING PUBLIC VALUES FOR COMPLEX POLICY DECISIONS Energy producl~on Tranapotlal~onand dislrlbullon cosls Long.mrm m s

r

-----+gsF1On rC O S01~ energy lnlensive producls

Financial lechnical

rTechnicalwlnerabll~ly

Technical reliabil~ly Ellic~encyol producllon process EH~ciencyol Ihe energy

requiremenls

Low malnlenance requiremenls

LLong lile

~ ~ ~ ~ ~ ~ : ~ ~ ~ ~ , " h " , n g e s ",:;'

Flex~b~lily Compal~biltly user-friendly

I

F Raw

Rducllon 01 imporf dependence ~ Sscur~ly ol energy suppltes

malerlals

~ tiNslsl,y

~

,

~

Technical disruplions Reduclbn ol polenlial disruptions lnlenlionai disruplions (eg sabolage) Adaplabil~lyol Vlct energy syslem Salislaclion ol economic n e e d s - - - ~ % $ % k s lo markel demand

Nal~onal

Produclivily Developmen\ol expotl markels

The economy's abillly 10 change

L%oin:

Adaplabilily

Local environmenl

Impacls on the natural environmn'

---EE,"Ek"i I a"as

Crealion and preservalion ol special natural environments

1

Preservationol species ~nhancemenI(egn c u ~ i v a l ~ o n ~

Regional anvironmanl

---

,

r

For individuals HealIh and d - wlely For miely

- -

Climale (eg via CO,) Impacl on llora and launa ol olher countries (eg acid rain) EThreals lo lire ThmaIs lo heallh Injuries Psvcholoqtcalimoacls Caiaslroihes . Long4erm lhreals l o manklnd lea oenellcl.

r

Cullural and moral qualily ol lile

Quality 01 lile

t

Social impacts

k i e t aw i s

Social qualily ol lile

-' . L ZL r

Enhance cullural bsources

Enhance cullural values (ea reverence.~ersonaland hiGorical penp&ive, lanlasy; reduce aggression. apalhy, unilwmily, waste. arlilicial consumplion) Enhance human Idenre (eqindividual developmenl o@ons, relalbnship lo nalure. reduce anonymily and angsl) r S & i juslice (eg opporlunily)

.--

Slable living

Economic qualily of lile

Energy services (eg heal. Ighl)

% ~ a E E ~ s For socio-cullural change

F;:divily

2; ~~~&

n,.piws,

lmprovemenl ol decision process aboul energy setem

runll

Willimness lo comoromise

Supporl by majorities

Persislence ol democralic decision making process

Political

~ ~ ~ $ lnSlllullOnS

%

%

r

~

~

d

Enhancemenl ol aulonomy

w r i w pea-

-c

impacts

Degree ol lreedom in inlernalional polilics-

z;

~

~

~

~

,

"

~ e e options p 01o h cwntnes n , National independence lnlernalionai cooperalion (eg unions. lirms, sclenlik organizations)

~

~

~

-z

~

1

~ ~ "~ ; i~ ~c~ i '~ t s$ $ l a l Blackmall polenlial

~~~~~~~ ::~~~~~~~~

FIGURE 1. Value Tree for Evaluating West German Energy Scenarios.

The qualitative characteristics of the four paths are briefly summarized in Table 1. As can be seen, they range from a relatively high energy growth path with a strong expansion of nuclear power and the introduction of the fast breeder (path 1 ) to a path with severe conservation and use of regenerative energy sources (path 4). In a separate study, Keeney, Renn and von Winterfeldt ( 1987) (see also Keeney, Renn, von Winterfeldt and Kotte 1984) structured the basic objectives and criteria by

~

~

~

1018


which the four paths should be evaluated. In interviews with leading representatives of 10 major German organizations, separate value trees were created that structure a hierarchy of general concerns (e.g., "Protect the environment"), intermediate objectives (e.g., "Minimize air pollution") and specific value relevant attributes of the energy paths (e.g., "Tons of SOz production per year"). These separate trees were logically combined into one tree with eight general objectives: Financial, technical and material requirements; security of energy supplies; national economic impacts; impacts on the natural environment; health and safety; social impacts; political impacts; international impacts. The complete value tree is shown in Figure 1. The Enquete Commission issued further studies to assess each path with respect to subsets of the main objectives shown in Figure 1. These assessments relied on facts and expert judgments to build the "factual basis" for the public value forum. In some cases, additional judgments by the leaders of the value forum led to specific reformulations of the expert assessments to fit the description of the objectives. The participants were selected from two groups of the general public that were thought to represent somewhat different views of the energy problem in West Germany: engineers and social science teachers. An invitation to participate in the public value forum was sent to two organizations: Engineering Societies and Associations of Social Science Teachers in the area of Bonn, West Germany. Nine engineers participated in the first value forum, and 14 social science teachers participated in the second forum. All participants were compensated for their time. The value forum took two full days. During the first day the tasks of the value forum were introduced, the objectives and attributes were discussed and refined, and singleattribute utility functions and tradeoffs were elicited. During the second day, the energy paths were introduced and evaluated, first intuitively, and then using the multiattribute utility model. During the last two to three hours of the second day, participants reconciled the discrepancies between the multiattribute utility models and their intuitive evaluations of the energy paths. Following are comments and specifications of the value forum for each of the six steps described in $2. 1. Introduction and Motivation. Participants were introduced to the forum staff (four members of the Nuclear Research Facility in Juelich and Keeney and von Winterfeldt), the forum agenda and the forum objectives. The forum leader emphasized that the main objective of the forum was to elicit value relevant information from a group of nonexperts and thus to test the feasibility of a procedure for obtaining public value inputs to the energy debate. The factual background on energy supply, supply mix and projections of future energy demand was presented by energy experts. The coal and nuclear options were discussed in some detail as was the effect of major conservation measures. The economic and technical constraints were identified and the possibilities for some novel technologies (e.g., solar and wind) were discussed. 2. ReJinement of Objectives and Attributes. Participants were introduced to the value tree in Figure 1 and they were told that they would be asked to evaluate the energy futures with respect to the eight main objectives in this tree. The whole tree was presented for reference to enable participants to understand the breadth and the meaning of each of these eight objectives (see Table 2). The forum leader encouraged participants to discuss and possibly modify the tree. For this purpose, small groups were formed to discuss parts of the tree. After the small group meetings, a final tree was created in a joint session. This tree was used throughout the evaluation. 3. Elicitation of Single-Attribute Utility Functions. Before eliciting single-attribute utilities, the participants were asked to provide an intuitive evaluation of the energy futures described in Table 3. The forum leader asked participants to carefully read the descriptions of the energy futures one future at a time and to form an overall opinion about their relative desirability, relative to the "bad" and "good" energy futures that


1019

TABLE 2 A Hypothetical Energy Future Objectives

Energy Future C

Financial, technical and material requirements

relatively low short and long term costs average material requirements stable development of costs

Security of energy supply

relatively high technical reliability little dependence on import no use of scarce resources relatively little adaptability

Impacts on national economy

slight improvement in employment rate; potential problems in the long run relatively high international competitiveness industrial production and development of social structure like before

Impacts on natural environment

less damages to environment than before; substantially decreased emissions; further reduction possible little use of land

Health and Safety

small threat to life and health by emissions and accidents catastrophes virtually impossible little .danger to future generations

Social impacts

relatively high standard of living (in the long run lower compared to B) "welfare society" with increased responsibility of the individual some minor negative consequences of industrialization

Political impacts

concentration of power like before increased reglementation of free market system small chance of limitation of civil rights

International impacts

average interdependency with foreign countries constant discrepancy between industrial and undeveloped countries misuse of energy technologies for war and terrorism impossible some financial aid for developing countries

created the reference points. After forming an opinion, they were to assign a rank of 1 to the "good" energy future and a rank of 6 to the "bad" one. The other energy futures were to be ranked in between from best to worst. Subsequently, participants were asked to assign a rating of 100 to the good energy future and a rating of 0 to the bad one. The participants were then asked to rate the remaining four energy futures on this scale. To construct single-attribute utility functions, the forum leader then handed out a form similar to Table 3 but asked participants to evaluate the energy futures separately on each objective. That is, each of the impact descriptions had to be rated on a scale from 0 to 100 where the end points of the scale were defined by the "bad" and "good" impacts on the respective objective. These single-attribute ratings defined the single

TABLE 3 Description of Energy Futures


Energy Future A

"Good Energy Future"

Objective P

P

low short term and long term cost minimal material requirements stable development of costs

r

short term use in costs and general inflation relatively high long term costs relatively high uncertainty about cost development relatively high depletion of resources and materials

r

high technical reliability strong dependence on import relatively high use of scarce resources average adaptability

r

Energy Future B

Energy Future C

low short term and long term costs minimal material requirements stable development of costs

relatively low short and long term costs average material requirements stable development of costs

very high short term costs low long term costs relatively high uncertainty about development of costs high depletion of resources and materials

high middle and long term costs substantial depletion of resources and materials high uncertainty about cost development

high technical reliability relatively little dependence on import no use of scarce resources high adaptability

relatively high technical reliability little dependence on import no use of scarce resources relatively little adaptability

low technical reliability little dependence on import no use of scarce resources average adaptability

low technical reliability strong dependence on import substantial use of scarce resources little adaptability of the energy system

unemployment rate relatively stable high international competitiveness small increase in industrial production leading to some imbalances in social structure

slight improvement in employment rate; potential problems in the long run relatively high international competitiveness industrial production and development of social structure like before

reorganization of job market 100%employment (autonomies or employment in industries) low international competitiveness little industrialization leading to balanced economic structure

increase in unemployment low international competitiveness increased concentration of industrial areas

less damages to environment than before; lower emissions; further reduction financially feasible little use of land

less damages to environment than before; substantially decreased emissions; further reduction possible little use of land

few damages to environment, but high material requirements; further reduction not feasible substantial use of land

substantial damages to natural environment with no technical or financial means to reduce these damages substantial use of land

Energy Future D

"Bad Energy Future"

--

Security of energ! supply

1 1

Impacts on national economy

1

high technical reliability little dependence on import no use of scarce resources high flexibility with regards to changes in supply, demand and cost

r

r

10090employment; high international competitiveness continuing industrialization leading to concentration of industrial areas

100% employment high international competitiveness balanced change in economic structure

r b

r

-

Impacts on natural environment

few damages to rural environment; impacts can be reduced by technical means little use of land

r

*

damages to natural environment which can be reduced by technical means average use of land

Health and Safet]

1

1

minor impacts on health catastrophes almost impossible little danger for future generations

B

B

B

Social impacts 1 1

high standard of living sufficient social security less anonymity and more involvement of individual citizens

B B

B

threat to life and health by emissions and accidents low probability of catastrophes possible danger for future generations

small threat to life and health by emissions and accidents very low probability of catastrophes little danger for future generations

high standard of living tendency towards "welfare society" negative consequences of industrialization (anxiety, isolation)

relatively high standard of living "welfare society" with some responsibility of the individual some negative consequences of industrialization

B

B

B

B

small threat to life and health by emissions and accidents catastrophes virtually impossible little danger to future generations

little technical risks but increasing natural risks catastrophes virtually impossible little danger for future generations

relatively high standard of living (in the long run lower compared to B) "welfare society" with increased responsibility of the individual some minor negative consequences of industrialization

relatively high standard of living. but low supply of consumer goods strong emphasis on individual responsibility (decentralization) possible negative consequences of a natureoriented lifestyle (dependency, spare time) -

Political impacts

1

democratic control of politics and society little regulation of the free market preservation of civil rights

r

B


1

1

1

Rank Rate

political independence from foreign countries less discrepancy between industrial and undeveloped countries misuse of energy technologies for war and terrorism impossible financial aid for developing countries

B

B

B

concentration of power in society, politics, and economy preservation of free market system limitation of civil rights possible in extreme cases

strong interdependency with foreign countries increasing discrepancy between industrial and undeveloped countries possible misuse of energy technologies for war and terrorism substantial financial aid for developing countries

rn

some concentration of powel in politics and economy some political reglementation in conformity with free market limitation of civil rights possible in extreme cases

relatively strong interdependency with foreign countries constant discrepancy between industrial and undeveloped countries possible misuse of energy technologyes for war and terrorism financial aid for developing. countries

B

B

B

B

concentration of power like before increased reglementation of free market system small chance of limitation of civil rights

average interdependency with foreign countries constant discrepancy between industrial and undeveloped countries misuse of energy technologies for war and terrorism impossible some financial aid for developing countries

B

B

substantial threat to life and health by emissions and accidents low probability of catastrophes danger for future generations

low standard of living social security cannot be guaranteed increase in anxiety, anonymity and isolation

-

complete decentralization of power in society and politics strong reglementation of free market system reglementation of life styles, but no limitation of civil rights

extreme concentration of power in politics, economy and society substantial control of citizens by the government limitation of civil rights that are guaranteed by the constitution

little interdependency with foreign countries less discrepancy between industrial and undeveloped countries misuse of energy technologies for war and terrorism impossible little financial aid for developing countries

high political dependence on foreign countries increasing discrepancy between industrial and undeveloped countries possible misuse of energy technologies for war and terrorism no financial aid for developing countries

1022


attribute utility functions. Subsequently, participants were asked to provide a new set of overall ratings of the four energy futures. 4. Elicitation of Tradeofi. First, participants provided a rank order of the eight main objectives in terms of their importance for evaluating energy futures. They were instructed to assign an importance rank of 1 to the most important objective and an importance rank of 8 to the least important one. They were encouraged to examine the overall value tree (Figure 1 ) to clarify the meaning of the objectives when making these judgments. The forum leader then pointed to a potential problem with the previous procedure since it ignores the ranges of the possible impacts of energy systems on the objectives. For example, many participants probably would rank "Environmental impacts" as one of the most important objectives. Yet, if all energy systems under consideration are very similar with respect to their environmental impacts, this objective should not be given a high weight since it does not discriminate between the alternatives. The session leader presented several examples to the participants to sensitize them to the problem of importance judgments. To alleviate this problem, the so-called swing method for weighting attributes was used (see von Winterfeldt and Edwards 1986). After some training in the swing weighting method, participants were presented with the "bad" and the "good" energy futures in Table 3. They were asked to carefully compare differences between the bad and the good impact across the eight objectives. Next they assigned a "swing rank" of 1 to that objective for which the change ("swing") from the bad to the good impact constituted the largest improvement, and a swing rank of 8 to the objective that constituted the least improvement. All other objectives received ranks in between. After providing such swing ranks, participants were asked to assign 100 points to the objective ranked 1, and to assign zero points to an objective, not necessarily one of the list, which they considered completely irrelevant. All other objectives were to be rated on that scale from 0 to 100. The resulting weights are the "raw" swing weights. 5. Construgtion of a Multiattribute Value Model and Evaluation of Alternatives. The raw swing weights of each individual were first normalized to add to one. Subsequently, an additive multiattribute value model was developed for each participant using the swing weights and the ratings of the impacts on each objective. At this time the four energy paths were introduced, which had identical impacts as the energy futures described in Table 3 (i.e., energy future 3 had identical impacts as energy path 3). In addition to describing the paths and highlighting their differences and similarities, four national experts gave presentations to advocate each path. Participants were then asked to evaluate the four energy paths with respect to each objective. No actual impact descriptions of the paths were provided. Thus, the ratings combined the participants' intuitive understanding of the "facts" as well as a relative evaluation of the perceived facts. Participants first ranked the paths on each objective and then rated them on a scale from 0 (bad impact on the objective) to 100 (good impact on the objective). To complete this session they provided an overall intuitive ranking and rating (0-100 scale) of the four paths. In addition, the forum organizers calculated a multiattribute utility evaluation of the four energy futures for each of the participants using the experts' impact assessments (Table 3 ) as well as the participants' ratings of these impacts and their weights. 6. Discussion and Resolution of Iizconsistencies. The forum leader explained to participants that the four energy futures evaluated during the third task were merely the descriptions of the four energy paths in terms of their impacts on the eight objectives. It was pointed out that this implies certain consistency requirements between the evaluation of the energy futures and the evaluation of the energy paths. The steps of the multiattribute utility analysis were explained in detail with an example. Participants were told that the results of their multiattribute utility analyses and their


1023

own intuitive evaluations probably showed some inconsistencies. The forum leader pointed out that these inconsistencies were nothing to worry about and they were also told that there were several reasons why these inconsistencies occur, including 1. an incomplete or inappropriate set of objectives and attributes; 2. an inappropriate aggregation rule for the multiattribute model; 3. biased intuitive evaluations of the futures or the paths; 4. biased weights; 5. biased single-attribute utility i h c t i m s (ratings); 6. biased expert assessments of the impacts. Participants were then handed out a form just like Table 3, except that now the futures were labelled as paths and the following additional information was provided: 1. the normalized swing weights; 2. the ratings of the energy futures on the eight component objectives; 3. the overall ratings of the energy paths; 4. the calculated multiattribute utilities of the energy futures (combining 1 and 2). Participants were asked to generate a final overall evaluation of the energy paths by refemng to the above described reasons for inconsistencies. If they considered the objectives or aggregation rule inappropriate (reasons 1 or 2), they were asked to qualitatively describe the desired changes or additions. If, however, they wanted to make adjustments to their numerical evaluations based on reasons 3-5, they should either change their weights, objective ratings, or overall evaluations. If they disagreed with the expert assessments of impacts, they should note desired changes in the appropriate cells of Table 3 and re-rate the impacts. Staff members assisted them in these tasks. The goal was to achieve at least ordinal consistency between the final overall evaluation of the paths and the multiattribute evaluation model of the futures. 4. Results All but two subjects participated in all tasks of the experiment. There were some minor modifications in the substructures of the value tree after the discussion session of the first day, but the general definitions of the objectives remained the same. Thus a common value tree could be developed for all participants. Participants found the evaluation tasks stimulating and in the end strived for a consistent set of judgments which sometimes involved three hours of reconciliation. Most subjects achieved ordinal consistency between their intuitive evaluations of the energy paths and the rankings derived from the multiattribute utility model. All subjects who participated in the reconciliation were able to align their intuitive first choice with the calculated first choice. As already indicated in the descriptions of the public value forum, the two day procedure produced a large amount of data for each participant. In addition to these raw data, we calculated three multiattribute utility models: 1. Futures Model: combines initial swing weights with ratings of the impacts of the futures on each objective; 2. Path Model: combines initial swing weights with ratings of the paths on each objective; 3. Final Model: combines final swing weights with revised ratings of the revised impact descriptions on each objective. The discussion of the results is organized as follows: First we will analyze the importance and swing weighting data. Second, we will analyze the overall ratings of paths and futures and their changes over the course of the value forum. Third, we will compare the overall ratings with the model results. For all analyses, we will begin with a description of the total sample, analyze individual trends and study differences between teachers and engineers.

1024

RALPH L. KEENEY, DETLOF VON WINTERFELDT A N D THOMAS EPPEL TABLE 4 Importance and Swing Weights

Objective

Median of Importance Rank

Mean of Swing Weights

Median Rank o f Swing Weights


5.0

0.08

7.50

Security of energy supply

4.0

0.1 1

5.50

Impacts on natural economy

4.0

0.13

4.75

Impacts on national environment

3.0

0.13

4.00

Health and Safety

1O .

0.18

1.75

Social impacts

3.5

0.13

3.25

Political impacts

5.O

0.14

3.5


7.0

0.10

5.75

Comparison of Importance Weights Table 4 shows the median of the individual importance ranks for the eight objectives as well as the mean swing weights for the total sample. Only one participant ranked the objectives identically in both methods. On average, there were six (out of a possible 28) rank reversals, the median of individual rank correlations across all subjects is 0.70. In general, the objectives concerned with "International Impacts" and "Political Impacts" received a higher weight in the swing weighting method. There was no difference between the weights of teachers and engineers. None of the differences between initial and final swing weights were significant. In addition, the preference for one of the four futures or paths did not correlate with special weighting patterns. Overall Evaluation of the Energy Futures and the Energy Paths Participants provided four overall ratings: initial energy futures, final energy futures, initial energy paths, and final energy paths. This section analyzes how their overall evaluations change in the course of the value forum, especially how they are affected by the information about the paths (futures-paths) and by the formal reconciliation process (initial energy path-final energy path). Furthermore, we will analyze how different attitudes towards energy technology, as represented by teachers and engineers, influence these evaluations. The average ratings for the four sets of ratings are shown in Table 5 both for the total sample and separately for engineers and teachers. The table suggests the following pattern. For the total sample the energy future C (moderate growth, conservation with gradual phasing out of nuclear energy) was favored. In contrast, the initial average path ratings favored path 2 (moderate growth with expansion of nuclear power). After reconciliation the overall ratings of the paths resembled those of the energy futures much more closely. Path 3 is now evaluated the highest and path 2 is a close second. Teachers and engineers provide very similar results, especially for the energy futures. A notable exception is again path 3, which engineers initially rated much lower than teachers, but engineers adjusted that rating substantially after reconciliation. Table 6 gives an indication of the intra-individual agreements between the four sets

'

ELICITING PUBLIC VALUES FOR COMPLEX POLICY DECISIONS TABLE 5 Intuitive Ratings (a) Intuitive Ratings ofNEnergyF~rtures" Total

Engineers

Teachers

Initial

Final

Initial

Final

Initial

Final

FA

23.6

31.1

22.8

37.1

24.2

26.9

FB

49.2

53.0

52.2

57.2

47.2

50.0

FC

73.2

69.2

68.3

68.6

76.6

69.6

FD

55.3

50.1

46.1

45.9

61.6

53.1

n

22

22

9

9

13

13

(b) Intuitive Ratings ofUEnergyPaths" Total

Teachers

Engineers

Initial

Final

Initial

Final

Initial

Final

P1

31.1

30.3

36.7

29.3

27.5

30.7

P2

66.0

64.6

63.2

62.1

67.9

65.7

P3

59.7

68.1

45.2

67.5

68.9

68.3

P4

36.3

44.6

34.3

53.7

37.5

40.4

n

23

19

9

6

14

13

of ratings. It shows the average number of rank reversals between all pairs of ratings. Relative to a maximum of six possible reversals per pair, the general trend indicates a fair amount of consistency across the four sets of ratings. The largest number of reversals occurs between the rankings of energy futures and energy paths, the smallest between the two futures ratings and the two path ratings. TABLE 6 Average Ntrrnber of Rank Reversals Between Overall Ratings Total Sample

Engineers

Teachers

Initial future vs. final future

0.96

1.38

0.69

Initial future vs. initial path

1.77

1.77

1.77

Initial future vs. final path

1.44

1.20

1.56

Final future vs. initial path

1.65

1.83

1.53

Final future vs. final path

1.4 1

1.83

1.38

Initial path vs. final path

0.45

1.14

0.15

1026


There does not appear to be a substantial difference between engineers and teachers. Two sets of numbers are, however, noteworthy: engineers tend to change their ranking between the initial and final evaluation of futures much more than teachers ( 1.38 reversals vs. 0.69) suggesting that the task of numerically evaluating the futures on each objective influenced engineers more than teachers. Similarly, engineers changed their rankings between the initial and final evaluation of paths more than teachers, suggesting a stronger influence of the numerical reconciliation process for engineers. Over the course of the value forum, an increasing number of participants expressed a preference for the more moderate alternatives (futures B and C, and path 2 and 3 respectively): 68% preferred either futures B or C, in the initial rating, 73% preferred either B or C in the final futures rating, 82% preferred either path 2 or 3 in the initial path rating, and in the final evaluation, 89% preferred either path 2 or path 3. To give an indication of the direction of these changes, some of the most interesting comparisons of Table 6 were cross-tabulated in terms of changes of the preferred alternative. Table 7a shows how participants changed their preference from the final futures evaluation to the initial path evaluation. The most common changes of preference is from future C to path 2. This change is partially reversed through the reconciliation process (Table 7b). It is interesting to note that only 3 1% of the participants prefer the same alternative at the beginning and at the end of the forum. The remaining participants changed from future C to path 2 (44%) or from future D to path 3 (25%). Both changes indicate a move away from low growth, conservation oriented futures.

Comparison of the Intuitive and Multiattribute Evaluations Table 8 gives the overall and the corresponding multiattribute evaluations at different stages in the value forum. It can be seen that the three models produce fairly similar TABLE 7 Final Futt~revs. Path 7a: Final Future vs. Initial Path Initial Preference for Path 1

Path 2

Path 3

Path 4

Future A

1

0

0

0

8 2

Future B

0

2

0

0

-E

Future C

0

8

5

1

G

Future D

0

1

2

2

CE crl

7b: Final Fullire vs. Final Path

Final Preference for

c.

'2

Q)

-

c

2 2 P,

CE

-

Path 1

Path 2

Path 3

Path 4

Future A

0

0

0

0

Future B

0

2

0

0

Future C

0

6

5

1

Future D

0

I

3

1

crl.

.d

L

ELICITING PUBLIC VALUES FOR COMPLEX POLICY DECISIONS TABLE 8 Overall and Culc~ilarcd(Mldriatrribure)E\wl~ruriorto/Flilwes artd Parhs Overall Ratings

Calculated Utilities

Final Future

Inital Paths

Final Paths

Futures Model

Path Model

Final Model

FA, P1

31.1

3 1.1

30.3

36.5

30.6

30.7

FB, P2

52.9

66.0

64.6

59.4

55.6

56.4

FC, P3

69.2

59.7

68.1

64.3

66.3

67.2

FD, P4

50.1

36.6

44.6

48.7

44.6

47.5

average multiattribute utilities. In addition, the average intuitive ratings at each stage correspond fairly well with the model built at that stage. An exception is the initial path rating, where the model clearly favors path 3 while the overall ratings tend to favor path TABLE 9 Preferences Infirred from Overall and Multiattribute Evaluations 9a: Final Future

Preference Calculated from "Futures Model" Future A

Future B

Future A

0

0

1

0

Future B

0

1

1

0

Future C

0

1

12

1

Future D

0

0

2

2

9b: Initial Path

Future C

Future D

Preference Calculated from "Path Model" Path 1

Path 2

Path 3

Path 4

Path 1

1

0

0

0

Path 2

0

7

3

0

Path 3

0

0

7

0

Path 4

0

0

0

3

9c: Final Path

Preferences Calculated from "Final Model" r

Path 1

Path 2

Path 3

Path 4

Path 1

0

0

0

0

Path 2

0

2

7

0

Path 3

0

0

8

0

Path 4

0

0

1

1

1028


2. This pattern repeats on an individual level. On the average, there is less than one reversal between the rank orders of model vs. overall evaluations. As far as preference for one of the alternatives is concerned, there is also significant agreement between the calculated and the intuitive preference. Table 9 shows cross-tabulations of the calculated vs. the intuitive preferences expressed by the participants over the course of the forum. As can be seen from Table 9a, calculated and intuitive preferences agree in 15 out of 2 1 (71%) for the final futures. This rate of agreement is even higher for initial paths (9b; 86%), but lower for final paths (9c; 58%). It should be pointed out that this high agreement is not due to the fact that preferences do not change over the course of the forum. Table 9 shows an initial shift towards a harder energy option between the final evaluation of energy futures (only 1 preference for futures A or B) and the initial evaluation of energy paths ( 8 preferences for paths 1 or 2). This shift is reversed in the final evaluation of paths (again only 2 preferences for paths 1 or 2). In most cases, these shifts in preference are indicated by both the intuitive and the calculated judgments.

5. Conclusions Returning to the four questions raised in the introduction, we can summarize the results of this study by stating that: ( 1 ) the overall process of a value forum is feasible and, in particular, values can be systematically elicited from nonexperts and combined with the factual inputs from experts; ( 2 ) values elicited formally disagree with values elicited informally, but these discrepancies can be resolved at least in an ordinal sense; (3) values elicited without specific knowledge of the policy alternatives disagree with values elicited with that knowledge, but such discrepancies can also be resolved; (4) the value forum can provide important value relevant information for the policy process and it is educational for participants. It is, however, time consuming and relatively expensive.

Feasibility oj'the Public Value Forum The first conclusion is based primarily on four observations: laypeople came to the public value forum, felt comfortable with the procedure, carried it out, and were eager to resolve inconsistencies. Only two participants did not fully cooperate. One participant announced his unwillingness to go through such an "esoteric numerical exercise" during the course of the forum. The other did not make such an announcement, but did not provide some of the answers required and did not participate in the reconciliation phase. Both participants were teachers. They both stayed throughout the forum and actually assisted some of the other participants in the reconciliation task. Most participants questioned the experts in direct exchanges, but few questioned the expert assessments in Table 3. In fact, there was a strong acceptance of the idea that their value judgments should be combined with the expert assessments provided in Table 3 to form an overall evaluation of the energy alternatives. Formal vs. Informal Value Judgments There clearly was a substantial amount of disagreement between formally and informally elicited values. Throughout the forum 80% of the participants changed their rank order of the alternatives and 67% changed their preference between the first evaluation of the futures and the final evaluation of paths. Apparently, the value forum process and the multiattribute utility modeling had a significant influence on their thinking. Perhaps the most striking impression that was left with the forum staff was the energy and diligence


1029

with which the participants worked at the reconciliation of their initial path evaluations with the "Initial Futures Model." They felt that these two sets of evaluations should agree, refused overwhelmingly the "easy solution" to reject the objectives or the model, and worked hard, up to three hours, to achieve ordinal consistency. In this reconciliation phase the primary changes were in weights and in intuitive evaluations.

Labelled vs. Unlabelled Alternatives We had hypothesized that an evaluation carried out purely in terms of the impacts of the alternatives (energy futures) would be different from the same evaluations, once the alternatives are labelled. In particular, we assumed that there would be more controversy and disagreements about the labelled alternatives, because people's preconceived notions of the alternatives might override their "detached" evaluation of the impacts. There was indeed a substantial difference between futures and path evaluations, primarily in a shift of preference between future C and path 2. The final resolution of that difference was in favor of path 3, as suggested by all three models. The explanation of this effect seems, however, fairly specific to the respondents and the alternatives. The energy path labels seem to have evoked very specific preferences, as the main source of the mistake were four teachers who initially preferred path 2 and future C, but later converged on a preference for path 3. The overall effectwas, however, a moderation in the overall evaluations throughout the forum as well as a convergence towards the rank orders suggested by the multiattribute utility models. This finding replicates several studies in decision analysis which indicate that the use of multiattribute utility procedures tends to encourage convergence and moderation (see, e.g. Gardiner and Edwards 1975; von Winterfeldt and Edwards 1986). Advantages and Disadvantages of the Value Forum Like the other methods for obtaining public value information described in the introduction, the public value forum is to provide information to policy makers. As this example shows, the value forum can, indeed, provide very specific and useful information, especially about the tradeoffs that participants feel appropriate for evaluating alternative policy options. The two particular value forums suggest, for example, that participants considered the range of health and safety impacts important for evaluating energy policies, but political, economic, and environmental criteria do not come far behind. The acceptance of the expert assessments by the participants and the relative flexibility in their weights and intuitive evaluations indicate that there exists much room for negotiation and adjustment on the value side of the energy policy problem. What is less often appreciated is the fact that the process of eliciting and reconciling value relevant information can lead to changes in the participants' evaluation as well. In this study, there clearli was a shift towards moderation on both sides of the energy debate as the forum progressed. Thus, another advantage of the forum is the educational effect that it may have on the participants. The greatest disadvantage is, of course, the time and cost of the forum. It took two professionals to design the forum and six to conduct it. Summing design, preparation, and session time, this amounts to approximately 60 person days. In addition, the time of the participants amounted to 46 person days. At an average cost per person per day of $300, it would cost over $30,000 to repeat these value forums. A related disadvantage is the small sample and the lack of representativeness. Considering the advantages and disadvantages of the value forum, the best use of a value forum might be for value elicitation and education with small groups of key represent~tivesand leaders of stakeholder groups involved in a specific policy debate. The information obtained and transmitted in these group sessions may naturally filter down

1030


through the s t a k e h o l d e r groups, or it c o u l d be f o l l o w e d up with a d d i t i o n a l q u e s t i o n n a i r e t y p e e l i c i t a t i o n s based on the v a l u e forum results.'

'

This study was carried out as part of a larger project on the social compatibility of energy systems in the Federal Republic of Germany (FRG). The experimental study was sponsored by the Nuclear Research Facility Juelich. Data analysis and the preparation of this report was funded by the National Science Foundation under grant No. SES-8520167. We thank the two referees, the Associate Editor and Robert L. Winkler for helpful comments on an earlier draft. The views and opinions expressed in this article should not be attributed to the sponsors of this study.

References BROWN,C., "The Central Arizona Water Study: A Case for Multiobjective Planning and Public Involvement," Water Resources Bulletin, 20 (1984), 331-337. CAMERON, T. A. AND M. D. JAMES,"Efficient Methods for 'Closed Ended' Contingent Valuation Surveys," Rev. Economic and Statist. , 69 ( 1987) , 269-276. CREIGHTON, J., J. DELLIPRISCOLI AND C. M. DUNNING, "Public Involvement Techniques: A Reader of Ten Years of Experience at the Institute for Water Resources," IWR Research report 82-R1, Institute for Water Resources, Ft. Belvoir, VA, 1982. DESVOGUES, W. H. AND K. V. SMITH,"Focus Groups and Risk Communication: The "Science" of Listening to Data," Risk Analysis, 8 (1988), 479-484. EDWARDS, W., "How to Use Multiattribute Utility Measurement for Social Decision Making," IEEE Trans. Systems, Man Cybernet., SMC-7 ( 1977), 326-340. . Zukuenftige Kernenergie Politik . Deutscher Bundestag, Bonn, 1980. ENQUETEKOMMISSION FISHBEIN,M. AND I. AJZEN,Belief, Attitude, Intention, and Behavior. Addison-Wesley, Reading, MA, 1975. GARDINER, P. AND W. EDWARDS,"Public Values: Multiattribute Utility Measurement for Social Decision Making," In M. F. Kaplan and S. Schwartz (Eds.), Human Judgment and Decision Processes. Academic Press, New York, 1975. KEENEY,R. L. AND H. RAIFFA,Decisions with Multiple Objectives: Preferences and Value Tradeofs , Wiley, New York, 1976. , 0. RENNAND D. VON WINTERFELDT, 'LStructuring West Germany's Energy Objectives," Energy Policy, August 1987, 352-362. --AND K. KOTTE,Die Wertbaumanalyse: Entscheidungshilfe fuer die Politik. High , , Tech Press, Munich, 1984. MERKHOFER, M., Decision Science and Social Risk Management, Reidel, Dordrecht, Holland, 1987. MISHAN,E. J., Cost-Benefit Analysis. Praeger, New York, 1976, RENN, O., U. STEGELMANN, G. ALBRECHT AND U. KOTTE, "The Empirical Investigation of Citizens' Preferences for Four Energy Scenarios," Technological Forecasting and Social Change, ( 1984))26. VON WINTERFELDT, D., "Value Tree Analysis: An Introduction and an Application to Offshore Oil Drilling," In P. Kleindorfer and H. Kunreuther (Eds.), Insuring and Managing Hazardous Risks: From Seveso to Bophal and Beyond, Springer, New York, 1987, 349-377. AND W. EDWARDS, Decision Analysis and Behavioral Research. Cambridge University Press, New York, 1986.

Creating Policy Alternatives Using Stakeholder Values Robin Gregory, Ralph L. Keeney

Management Science, Volume 40, Issue 8 (Aug., 1994), 1035- 1048. Stable URL: http://link~.jstor.org/sici?sici=0025-1909%28 199408%2940%3A8%3C1035%3ACPAUSV%3E2.0.C0%3B2- 1

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at http://www.jstor.org/about/terms.html.JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content in the JSTOR archive only for your personal, non-commercial use. Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed page of such transmission.

Management Science is published by INFORMS. Please contact the publisher for further permissions regarding the use of this work. Publisher contact information may be obtained at http://www.jstor.org/journals/informs.html.

Management Science 01994 INFORMS JSTOR and the JSTOR logo are trademarks of JSTOR, and are Registered in the U.S. Patent and Trademark Office. For more information on JSTOR contact [email protected]. 02002 JSTOR

http://www.jstor.org/ Fri Nov 22 165451 2002