Using Evaluation Findings for Policy: A Realist Perspective Gary T ...

Download PDF
2 downloads 80 Views 50KB Size Report
Comment
1Based on Mark, Melvin, Gary T. Henry, and George Julnes. 2000. ..... (Ivengar, Peters, and Kinder, 1982, Krosnick and Kinder, 1989, Krosnick and. Brannon ...
Using Evaluation Findings for Policy: A Realist Perspective 1 Presented at

2000 European Evaluation Society Conference by

Gary T. Henry Andrew Young School of Policy Studies Georgia State University Atlanta, Georgia

Abstract

The author of this chapter argues that pursuing use as the defining goal for evaluation can distort the allocation of evaluation resources and reduce the contributions of evaluation to broader social goals, such as social betterment.

1

Based on Mark, Melvin, Gary T. Henry, and George Julnes. 2000. Evaluation: An Integrated Framework for Understanding, Guiding and Improving Policies and Program. San Francisco: Jossey Bass. 1

Why Not Use? Why not use as the defining goal for evaluation? After all, evaluation is a practical field and what could be more practical than use? Like the mighty Percheron whose strong legs and back were replaced by a variety of machinery, without use evaluation will be let out to pasture and preserved only to provide perspective on the past. The use or utilization of evaluation has reached a vaunted place supplying both a subject of study and a guiding purpose for evaluators. It is the latter sense, in which use is embraced as the Holy Grail of evaluation, which I will address in this chapter. However, the study of use can slide into the same precipice as the pursuit of use, when it ventures into prescription. A series of studies in the 1970s exposed the Achilles heel of policy and program evaluations – that the evaluations were not being used directly (Patton, Grimes, Gutherie, Brennan, French, and Blyth, 1977, Knorr, 1977, Caplan, Morrison, and Stambaugh 1975, Weiss and Bucuvalas, 1977). Subsequently, these insights were formulated into numerous strategies and theories to shield evaluation’s vulnerable heel (Bryk 1983, Weiss, 1983, Nachmias and Henry 1979, Patton 1978). These prescriptions sought to turn weakness into strength, making use into the criterion by which the success of an evaluation would be defined. The most fully developed and articulated of these theories, first published in 1978 and significantly refined in later editions, became an influential, if not the most influential theory of evaluation in the latter part of the century: Utilization-Focused Evaluation begins with the premise that evaluations should be judged by their utility and actual use; therefore, evaluators should facilitate the evaluation process and design any evaluation with careful consideration of how everything that is done, from beginning to end, will affect use (Patton, 1997, p. 20, emphasis in original). Use crystallized the purpose for evaluation and “intended use by intended users” became the mantra for a league of evaluators inspired by Michael Quinn Patton through his powerful and persuasive writing and through his extensive practice. Patton not only championed focusing on use, but, from the vantage point of a participant-observer, expanded evaluators’ awareness of types of use, including process use and developmental use (1997). Use of evaluation is, without question, an important consideration if evaluation is to escape the plight of the Percheron, but should it be the defining goal? Should evaluators set out with the goal of use as the criterion by which they judge the success of their work? If not use, what? In this chapter, I will begin by taking up the last question first and proposing an alternative goal for evaluation, the goal of social betterment. I will then show that evaluation priorities are affected by the selection of one defining goal for evaluation over the other. Then, I will attempt to explain the way that the goal of utilization traps evaluators, a phenomenon that I refer to as the paradox of persuasion. Persuasion is a specific form of use that is vital for evaluation to lead to betterment, but if one begins an evaluation with the goal of persuasion, then the evaluation looses the credibility necessary to persuade. Finally, since the road to

2

social betterment is long and arduous, I will close by offering more proximate outcomes to assess the performance of evaluations. Social Betterment The distinction that I propose to make may, at first, seem superficial and strained: the goal of evaluation is not use, the goal is social betterment. Use, then, while not the ultimate goal is a means by which evaluation achieves social betterment. However, not all use automatically produces social betterment and not all forms of social betterment can be achieved by pursuing use. Social betterment means improved social conditions, the reduction of social problems, or the alleviation of human distress (Henry and Julnes, 1998, Mark, Henry, and Julnes, 2000). For evaluation findings to contribute to social betterment, they must somehow be used. But use is a means or process through which social betterment can occur. The evaluation enterprise is no less vulnerable to goal displacement than the interventions that evaluators study. As Caplan pointed out in his assessment of Head Start, the “original goals were far more broad based than allowed for by the scope of the measuring instruments…evaluators lost sight of the original intent of the programs, and soon issues of race and intelligence replaced questions of emotional deprivation, sociological ‘damage,’ and educational achievement” (1977, p. 191). Use, once injected as a goal or guidepost for planning an evaluation, can begin to take on a life of its own, rather than serving as a means to an end. In this way, use can become the defining goal of evaluation and, in the process, displace and even obscure the broader purposes of social betterment, improving social conditions, and alleviating human distress and suffering. In their efforts to provide information that will get used, evaluators can lose sight of what information is needed to inform the discourse leading to social betterment. While it is possible to make this distinction, is it, in practice, trivial? Hardly! The goal of evaluation is to promote social betterment, albeit indirectly by providing information to those charged with setting social policies, to those charged with carrying them out, to stakeholders including service consumers, and to citizens. Whether social conditions actually improve depends on what is done with the information and on the quality of the findings (and, perhaps, on good fortune). Pursuing social betterment requires that three functions be performed: 1. Determining what constitutes the “common good”; 2. Choosing a course of action leading to the common good; 3. Adapting the chosen course of action to specific circumstances. Each function, as I will point out, requires different information, which corresponds to different priorities for evaluation. Determining the “Common Good”. More often than not, we define the common good by what it is not rather than by what it is. Take President Roosevelt’s depression era rallying cry that “One third of the nation is ill-fed, ill-

3

clothed and ill-housed.” While we could establish the principle that all members of society should be adequately fed, clothed, and sheltered, as a society we can reach consensus more readily about the imperative to confront the absence of these conditions than about the conditions that we expect to achieve. For instance, is someone adequately housed when sleeping in a shelter that is free from theft and violence? Or is it adequate to sleep over a heating vent when the individual involved chooses that over a shelter? Kendler (1999) notes that, “Human cognitive ability is so flexible and creative that every conceivable moral principle generates opposition and counterprinciples” (p. 832). Democracies in which their members have the autonomy to express their views are uniquely suited to the generation of oppositional statements and value pluralism is the inevitable result (Galston, 1999). Therefore, defining the “common good” is usually a matter of defining a need or problem around which there is sufficient consensus that it is considered a social problem or, using Berlin’s terminology, a “human need” (1998). Evaluations can support and inform the determination of the “common good” by providing an empirical check on the claims of need and the extent or consequences of a social problem. In the field of public health, surveillance studies often supply information on changing patterns of illness and causes of death. Many states have developed performance measurement systems for public education that report on average levels of achievement and educational progress. Other evaluations may point up the fact that social problems are persisting in the face of existing social policies to reduce them. Or, that some groups are not benefiting from the existing policies, thereby indicating an unmet need. These types of evaluation are initiated both from within and from outside democratic institutions. In Georgia, for example, there are no fewer than four groups that compile and report on school performance. One is done by the state agency that is responsible for funding and supporting public schools. Another is produced by an independent agency of the state overseen by an appointed council. The state’s largest newspaper and a conservative, business funded foundation each offers its own school report cards. Foundations with social missions, such as the Annie E. Casey Foundation through the support for the state-by-state publication of Kids Count, and policy advocate organizations such as Mother’s Against Drunk Driving, marshal evaluative information in an attempt to persuade journalists and policymakers that a specific social problem deserves a place on the policy agenda. Other evaluations that empirically underscore social problems are those that show the ineffectiveness of current policies. While evaluations that produce these types of findings are not planned to produce negative results they often raise questions about the existing service routines and open the door for policy entrepreneurs to offer alternative policy solutions. For example, the well known “Fort Bragg study” rigorously evaluated a highly touted “continuum of care” regime for providing mental health services and found little effect compared to traditional methods for provision of services (Bickman 1996). Thus, the evaluation findings confronted the existing dogma about the efficiency and effectiveness of the “continuum of care” approach, but did not

4

provide any information that could guide the selection of an alternative regime that might improve the mental well being of the service consumers. Selecting a Course of Action. Evaluations can assist those in democratic institutions and ordinary citizens in assessing alternative courses of action that can be adopted as policies and programs. Rather than providing an empirical check on the claims about the social problem, these evaluations provide information on whether one course of action alleviates the problem better than another. For example, Gueron (1997) describes the welfare reform evaluations that tested the effectiveness of work requirements in direct comparison with existing programs. The reforms resulted in reduced welfare roles, more employment, and higher wages when compared with the existing system. By providing a head-to-head comparison of competing alternatives, these evaluations contributed to the selection of reforms by Congress, the administration, and state legislatures. In a similar fashion, the original High/Scope evaluation pitted the very highly structured, child-centered model against more eclectic nursery school approaches (Weikhart and Schweinhart, 1997). The results show improved social outcomes for the High/Scope participants in comparison to those who attended the less structured program. The evaluation has been influential in establishing preschool programs for children throughout the United States. The relative success of states such as Kentucky, Texas, and North Carolina that have implemented high stakes accountability systems on the National Assessment of Educational Progress, has tipped the scales toward high stakes accountability in many other states. While evaluations that influence policy choices generally involve cooperation with government agencies, they are by no means always conducted or sponsored by them. Foundations can supply the resources to operate and rigorously evaluate pilot programs. The evidence of success or failure can be used to persuade policymakers about the efficacy of a pilot program as a policy solution. In addition, while not serving the overarching goal of betterment, private organizations often seek to evaluate policy and program options as well as training and products in head-to-head comparisons. Whether seeking information on how a new soft drink will stack up against a competitor or whether a marketing approach enhances a corporate “bottom line,” evaluations designed to provide information on the selection of one course of action or another are common to the worlds of business, government, and not-for-profits. What may differ is the frequency with which alternative courses are actually considered in the private sector relative to the public sector. Even rigorous evaluations that directly compare one course of action to another support rather than supplant making the selection. One of the most important findings in the study of value pluralism is that it is impossible to yield a monotonic rating of the weights attached to various outcomes or effects across diverse stakeholders (Galston, 1999). Evaluations of this sort require a focus on some of the valued outcomes, but inevitably leave out some process measures or outcomes that some citizens or stakeholders value. Moreover, the evaluations are fallible and the extrapolation from one setting and time to another requires an

5

inferential leap. Evaluation information should lead to better decisions by democratically authorized bodies. Rather than attempting to make these judgements, evaluators are on firmer ground by informing citizens and policymakers about the likely consequences of various courses of action (Kendler, 1999), thus providing a check on the excesses of political rhetoric. Adapting the Course of Action. Evaluation can assess the strengths and weaknesses of the implementation of a chosen course of action and organizational as well as situational barriers to implementing the action. What can we expect from a state sponsored, High/Scope preschool, if the curriculum and instructional strategies are not understood or utilized by the teachers? Can we expect the benefits from welfare reform when the requirement to work is added but the resources and programs for finding jobs are not made available? Should a program that has been successful in lowering dropout rates for inner city African American youth be modified for suburban Latinos and Latinas? What if the implementation of the treatment regime for hospitalized mental health service consumers results in rates for the use of restraints that are twice as high as similar states? Evaluations can raise these issues and supply important information for those responsible for making policies and programs work. Evaluability assessments (Wholey, 1994, Rutman, 1988), program templates (Scheirer, 1996), logic models (Funnel, 2000) are all attempts to systematically provide feedback to adapt the chosen course of action or to adapt the organization responsible for administering the action for better results. Evaluation Priorities To achieve social betterment, collectively we must decide what are the most pressing social problems, what actions should be taken to address them, and continue to refine the actions that are taken and re-engineer the organizations that are charged with achieving the outcomes. However, rather than neat and linear, all of these processes co-exist at once and are more or less current in policy and administrative deliberations (Hilgartner and Bosk, 1988). Unfortunately, the resources for evaluation are limited and not all issues can be evaluated simultaneously with a sufficient degree of rigor. There’s the rub, the first spot where placing a priority on utilization versus social betterment begins to chafe. A clear focus on social betterment would use an assessment of the policy context to allocate evaluation resources (Mark, Henry and Julnes, 2000). When competitors to the existing service delivery system arise, they would be evaluated so information could be available to inform deliberations. As advocates press for governmental responses to needs that have not previously been addressed, the needs would be assessed. When a course of action has been firmly put into place and receives wide support, both improvement-oriented evaluations and those designed to detect omissions or other problems would be carried out. However, with use as the goal, the perspective narrows. Rather than assessing the policy context, evaluators looking for the most immediate

6

opportunities for use will be drawn to studies that focus on adapting programs or improving organizations. For over thirty years since the term incrementalism was coined (Lindbloom, 1968), we have realized that small rather than abrupt changes characterize the policy environment. Some research shows that we have drifted even further toward incrementalism since then, in spite of recent trends toward divided governments (Jones, True, and Baumgartner, 1997). New definitions of social problems do emerge, but they are also relatively rare and seem to accrete rather than spring dramatically onto the public agenda (Nelson, 1986; Glazer, 1994; Scarr, 1998; Habermas, 1996). Thus, neither new social problems nor new courses of action present frequent opportunities for evaluation to be directly useful. If we focus on use, either instrumental or developmental, resources will be concentrated on evaluations that support the third of the three functions required for social betterment, that is, adapting the course of action. The over-concentration on adapting or improving programs or organizations will occur simply due to the prevalence of these activities relative to the sparing opportunities for the other two (documenting a social problem and selecting a course of action). Because of the multiple interests and actors involved, no one manages the emergence of new problems on the public agenda and no one has a monopoly on determining the next, new policy alternative to be debated (Zhu, 1992). The judgments about which “human needs” make it onto the policy agenda and the deliberations through the media (Page, 1996), through elected bodies (Fishkin, 1995, Majone, 1989), and among the public (Page and Shapiro, 1992, Monroe, 1998, Stimson, 1998) will be ill informed if evaluation resources are not expended to inform deliberations about social problems and matching the problems with proposed solutions. Yet, most of the time, the issues for which evaluation can most immediately be used are in adapting policies, programs, and the organizations that administer them. In a similar vein, the stakeholders who can sponsor or otherwise commit themselves to an evaluation are likely to have ongoing roles with existing policies and programs, and to find the greatest utility in evaluation as an aid to management decisions. Use as the defining goal is likely to distort the allocation of evaluation resources toward administrative or organizational issues and away from social conditions and policy issues. The most immediate opportunities for use steer the commitment of resources in this direction and give short shrift to the information needed to support the other types of deliberations. But, is this merely a problem of providing balance along the temporal dimension of use (Kirkhart, Chapter One)? In other words, can we maintain the goal of use if we do a better job of marketing the long-term, less frequent functions? Evidence from examining one type of use oriented evaluation, evaluability assessments, strongly suggests that when the findings about organizational or program improvements are available, the interest in evaluation to support the other functions of betterment are not pursued (Rog, 1985). For example, once a program or organizational improvement oriented evaluation has been conducted and the improvements have been proposed, short shrift will be given to assessing the program’s merit or worth. But not only are other types of evaluation a tough sell on the basis of use,

7

directly pursuing use can compromise the evaluation, a situation which I refer to as the paradox of persuasion. The Paradox of Persuasion Persuasion is one way of labeling the somewhat inconsistently defined third category of use (Kirkhart, Chapter One; Shulha and Cousins, 1997). Unfortunately, persuasion as a type of use was sullied by its early connections with legitimation and symbolic use (Knorr, 1977): …social scientists’ data and arguments are used selectively and often distortingly to publicly support a decision that has been taken on different grounds or that simply represents an opinion the decisionmaker already holds (171-172). While she did not find evidence of the preponderance of this sort of use that ideologues expected, persuasion seemed to be the seamy side of use relative to instrumental or enlightenment uses. Majone (1989) admirably rescued persuasion from this connotation in a manner that is directly relevant to the pursuit of betterment through democratic institutions: To decide, even to decide correctly, is never enough in politics. Decisions must be legitimated, accepted, and carried out. After the moment of choice comes the process of justification, explanation, and persuasion. Also, policymakers often act in accordance with pressures from external events or the force of personal convictions. In such cases arguments are needed after the decision is made to provide a conceptual basis for it, to increase assent, to discover new implications, and to anticipate or answer criticism (p. 31). Majone (1989), as Campbell presaged (1982), recognized the deficiency of pursuing evaluation as a formal demonstration and shifted away from this moribund pursuit to that of evaluation as a tool for argumentation. The evidence from evaluation, by itself, does not conclusively demonstrate the choice that should be taken, but it provides justification and support for the existence of a problem or a course of action. Shulock (1999) creatively sought evidence of the persuasive use of evaluation and policy analysis by compiling data on the number of citations of reports of evaluation and analyses in the committee reports filed by Congressional committees after their deliberations concerning specific bills occurred. She finds that members of Congress are most likely to use citations as means for justifying and persuading others about their chosen course of action when committees were competing for jurisdiction and when public interest in the issue was high (p. 237-238). Thus, in the constant flow of potential problems and proposed solutions (Cohen, March, and Olsen, 1982; Kingdon, 1995), evidence from an evaluation enters into an existing soup of values, beliefs, preferences, and needs. The evaluation can be used to persuade members of the policy community and public, to justify a course of action, or to bulwark an argument. It cannot formally demonstrate or sufficiently determine what should be done (Majone, 1989, p. 22).

8

Here is the point where the paradox of pursuing use for the purpose of persuasion becomes inescapable. When the evaluation is planned, designed, and executed to facilitate persuasion as the particular form of use, the evaluator must begin with the objective of producing persuasive evidence. The endeavor becomes not evaluation, but “pseudo-evaluation” or an exercise of political advocacy, a well-recognized but soundly discredited approach to evaluation (Stufflebeam, 2001). Rather than providing a base of credible evidence that attempts to reduce the bias in selecting, collecting, analyzing, and reporting data, the evaluation picks and chooses ways to support the intended users’ preferred policy positions if persuasion is the goal. Pursuing persuasive use can lead us away from social betterment, if the information is revealed only when it justifies the course of action or importance of a problem. Evidence of program failures or that show that a problem may be less consequential than originally thought can be dropped as not supporting use, if use is the pursuit. Evaluators become advocates for a position, not advocates for raising the importance of information in public deliberations, in this case, information that has been as minimally biased as their skills and the study context permit. Let me be clear about what I am and am not arguing on this point. I am arguing that if social betterment is the defining purpose for evaluation our efforts to identify and select criteria to judge program success, our efforts to collect evidence based on those criteria, and our reporting of that evidence is motivated and justified by the possibility of making things better. Evaluations that proceed with this as an ultimate goal can use principles such as complete disclosure of information, and the requirement for information to “debunk bad ideas” within modern democracies (Hurley, 1989). I am not arguing that use as a defining goal corrupts a particular evaluation or evaluator. Indeed, as I indicated at the beginning of this chapter, use can and has proven to be a sufficiently proximate surrogate for social betterment that both lead in the same direction, however, they are not the same. With enlightened intended users and with evaluators who possess unswervable moral compasses, use is a reasonable goal. But as Madison informed us in the Federalist papers, we do not have a government by angels. In situations with “unenlightened” intended users, or with evaluators who are pressured to compromise the evidence, use can be served to the detriment of social betterment. Usually, social betterment or the larger social good supply the criteria for distinguishing the use from misuse. In these situations, at least when I have been in such situations, it is always the allegiance to betterment, the commitment to making things better for children or for service consumers, the commitment to democratic processes that provides guidance about what should be done. These are the arguments raised to defend openness and release of evidence that can be compelling to an identified user who is adamant about withholding certain information. The question of how society is best served, while there can be clear disagreement about the answer, pervades the dialogue for guiding every phase of an evaluation. The question of use is secondary to this, although essential to the discussion, as means to the ends. If use, especially persuasive use should not be the defining goal for an evaluation, how can we define immediate or proximate outcomes from an

9

evaluation that can lead to the long-term goal of social betterment? For a clue to answering this question, I will turn to the literature on agenda setting and the way that focusing events have been shown to influence opinions about what should be done. Letting Go of Persuasion and Raising the Salience of Program Effects Persuasion has been the focus of study for many fields and disciplines. After continued efforts to measure the impact of the media in terms of its ability to persuade the public regarding the issues of the day, Cohen (1963) summed up the situation this way: The mass media may not be successful much of the time in telling people what to think, but the media are stunningly successful in telling their audience what to think about (p. 16). A massive amount of evidence has been accumulated to document this assertion beginning with McCombs and Shaw (1972) and resulting in hundreds of published research articles. This literature that has been labeled as public agenda setting finds strong support for the “link between issues as portrayed in mass media content and the issue priorities of the public” (Kosicki, 1993, p. 101). The media are effective in influencing what issues the public think are most important or are most concerned about. In raising the salience of certain issues, the media influence the standards by which the public evaluates the performance of the government or public officials, through a process known as priming (Ivengar, Peters, and Kinder, 1982, Krosnick and Kinder, 1989, Krosnick and Brannon, 1993). The rather circuitous route by which the media influence public opinions may provide an analogy for how evaluations can foster persuasive use that leads to social betterment. Rather than pursuing persuasion directly, evaluations can be more effective by raising and documenting the effects of policies and programs for the public, program staff and administrators, and policymakers to consider. The effects can be processes, such as providing high quality childcare, or outcomes, such as improving readiness for school. They can be intended objectives or side effects. The more proximate indicators of the success of evaluations in the chain of events leading toward social betterment would include raising an issue from the evaluation findings for public deliberation. Or it could include providing an unbiased representation of the social problem or the ramifications of the policy that could stimulate “a shared understanding of the issue under discussion” (Majone, 1989, p. 6). Evaluations can contribute to the public understanding of an issue and influence the competition for framing the relevant issues of the day in very important ways by answering questions such as: Is the program working? How? For whom? In what circumstances? Providing a basis of facts that relate to valued effects or outcomes for public deliberations is no mean feat. In this process, the criteria used in the evaluation become more relevant and more influential in the informal judgments about the policy or program. The public and stakeholders alike are primed to use the criteria in reaching

10

judgments. While they are primed to make judgements influenced by the information, the evaluator needs to be aware that the criteria used in any evaluation do not exhaust the criteria that some citizens and stakeholders will consider important for reaching a judgment. Value pluralism in modern democracies and the generation of counterprinciples for every principle being offered will thwart excessive hubris from evaluators, policy entrepreneurs, or politicians in this regard. This would endorse the two-step process for the persuasive use of evaluation presented by Kendler: “The first step is to provide empirical evidence not contaminated by political goals. The second step is to avoid interpreting the results as directly justifying a particular social policy” (1999, p. 833). Conclusion Use has been evaluation’s Holy Grail, the anointed purpose, which has motivated a legion of evaluators. By reacting to the issue of non-use and rendering this Achilles heel of early evaluations less vulnerable, our work has escaped irrelevancy. Programs have been improved and organizations have learned much from evaluations focused on utilization. Evaluation, especially in the sense of evaluation as a professional field of practice, has not suffered from lack of relevance but perhaps from too much focus on the short run variety of use. Modern democratic societies clearly need information that supports the process of selecting one course of action over another and assesses the extent and severity of social problems, even though the uses for this information are less frequent and episodic rather than ongoing. This information is critical to our efforts toward social betterment or ameliorating social distress. However, questioning the assumption that the policy is the correct one for solving the problem or that the problem is significant are not as likely to occur to those participating as intended users, be they policy advocates or program administrators. If they do occur, they may be dismissed because of their “insider” biases. However, those in democratic institutions evidently sense the importance of this information because they call for it, although it seems more often from economists, sociologists, environmental scientists, and other non-evaluators, and their agents use it in developing and supporting their arguments (Shulock, 1999). But information useful for persuasion within these institutions will not be forthcoming from evaluators if they decide to pursue providing persuasive information directly. Rather, social betterment should be viewed as the ultimate purpose and raising the importance of policy effects in making policy judgments as a more immediate outcome for an evaluation. Results of these evaluations may achieve a circuitous route to persuasive use, raising salience of issues and contributing to the stock of information available to influence judgments about the consequences of social policies. Openness in providing information, adding public disclosure through the media and Internet to the task of private disclosure through reports and briefings, and participating in the discourses of the relevant policy community on the social problem and its alternative solutions are potential

11

roles through which evaluators can achieve relevance. These are the more proximate outcomes of evaluations that follow the route toward social betterment and the reliance on democracies, over the long haul, to debunk bad ideas when good information is made available. Relevance need not be sacrificed by replacing use with social betterment. Pursuing use generally will lead to an overemphasis on organizational and program improvement oriented evaluations. Instead the purposes of assessment of merit and worth, compliance and oversight, and knowledge development should receive their fair share of evaluation resources (Mark, Henry, and Julnes, 2000). Social betterment opens the field to a more deliberate selection of purposes, using the policy environment as a guidepost. Ultimately, the potential of evaluation is more likely to be realized if informing rather than influencing policies and programs is the criterion for success.

12

References Berlin, I. (1998). “My Intellectual Path.” New York Review, 45, (8), pp. 53-60. Bickman, L. (1996). “A Continuum of Care: More Is Not Always Better.” American Psychologist, pp. 689-701. Bryk, A. S. (ed.) (1983). Stakeholder-Based Evaluation. New Directions for Program Evaluation, no. 17. San Francisco, CA: Jossey-Bass. Campbell, D. T. (1982). “Experiments as Arguments,” in E. R. House, S. Mathison, J. A. Pearsoll, and H. Preskill (eds.), Evaluation Studies Review Annual, 7, pp. 117-127. Caplan, N. (1977). “A Minimal Set of Conditions Necessary for the Utilization of Social Science Knowledge in Policy Formulation at the National Level,” in C. H. Weiss (ed.) Using Social Research in Public Policy Making, Lexington, MA: Lexington Books, pp. 183-198. Caplan,N., Morrison, A., and Stambaugh, R. J. (1975). The Use of Social Science Knowledge in Policy Decisions at the National Level. A Report to Respondents. Ann Arbor, MI: The Institute for Social Research, The University of Michigan. Cohen, B. (1963). The Press and Foreign Policy. Princeton, NY: Princeton University Press. Cohen, M. D., March, J. G., and Olsen, J. P. (1982) “A Garbage Can Model of Organizational Choice.” Administrative Science Quarterly Fishkin, J. S. (1995). The Voice of the People: Public Opinion and Democracy. New Haven, CT: Yale University Press. Funnel, S. (in press, 2000). “Developing and Using A Program A Theory Matrix for Program Evaluation and Performance Monitoring,” in A. Petrosino, P. Rogers, T. A. Huebner, and T. Hacsi (editors) Program Theory Evaluation: Surveying the Field of Practice. New Directions for Evaluation, no. 87. Galston, W. A. (1999). “Value Pluralism and Liberal Political Theory.” American Political Science Review, 93 (4), pp. 769-778. Glazer, N. (1994). “How Social Problems are Born.” The Public Interest, no. 115, pp. 31-44. Gueron, J. M. (1997). “Learning About Welfare Reform: Lessons from Statebased Evaluations,” in D.J. Rog & D. Fournier (eds.), Progress and Future Directions in Evaluations: Perspectives on Theory, Practice, and Methods. New Directions for Evaluation, no. 76, pp. 79-94. Habermas, J. (1996). “Three Normative Models of Democracy,” in Seyla Benhabib (ed.), Democracy and Difference: Contesting the Boundaries of the Political, Princeton, NJ: Princeton University Press, pp. 21-30. Henry, G. T. & Julnes, G. (1998). “Values and Realist Evaluation,” New Directions for Evaluation 78, pp. 53-72. Hilgarten, S. & Bosk, C. (1988). “The Rise and Fall of Social Problems: A Public Arenas Model.” American Journal of Sociology, 94(1), pp. 53-78.

13

Hurley, S. (1989). Natural Reasons: Personality and Polity. New York, NY: Oxford University Press. Ivengar, S., Peters, M. D., and Kinder, D. R. (1982). “Experimental Demonstrations of the ‘Not-So-Minimal’ Consequences of Television News Programs.” The American Political Science Review, 76, pp. 848857. Jones, B. D., True, J. L., and Baumgartner, F. R. (1997). “Does Incrementalism Steam From Consensus or From Institutional Gridlock? American Journal of Political Science, 41, pp. 1319-1339. Kendler, H. H. (1999). “The Role of Value in the World of Psychology.” American Psychologist, pp. 828-835. Kingdon, J. (1995). Agendas, Alternatives, and Public Policies, (2nd ed.) New York, NY: Harper Collins College Publishers. Knorr, K. (1977). “Policymakers’ Use of Social Science Knowledge: Symbolic of Instrumental?” in Carol Weiss, (ed.) Using Social Research in Public Policy Making, Lexington, MA: DC Health & Company. Kosicki, G. M. (1993). “Problems and Opportunities in Agenda-Setting Research.” Journal of Communications, 43(2), pp. 100-127. Krosnick, J. A. and Kinder, D. R. (1989). “Altering The Foundations of Support for the President Through Priming.” American Political Science Review, 84(2), pp. 497-512. Krosnick, J. A. and Brannon, L. A. (1993). “The Impact of the Gulf War on the Ingredients of Presidential Evaluations: Multidimensional Effects of Political Involvement.” American Political Science Review, 87(4), pp. 963975. Lindblom, C.E. (1968) The Policy-Making Process. Englewood Cliffs, NJ: Prentice-Hall. Majone, G. (1989). Evidence, Argument, and Persuasion in the Policy Process. Yale University Press. Mark, M.M., Henry, G. T., and Julnes, G. (in press, 2000). Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs. San Francisco, CA: Jossey-Bass. McCombs, M. E. and Shaw, D. (1972). “The Agenda-Setting Function of Mass Media.” Public Opinion Quarterly, 36, pp. 176-185. Monroe, A. (1998). “Public Opinion and Public Policy, 1980-1993.” Public Opinion Quarterly, 62(1), pp. 6-27. Nachmias, D. and Henry, G. T. (1979). “The Utilization of Evaluation Research: Problems and Prospects,” in David Nachmias (editor), The Practice of Policy Evaluation, New York, NY: St. Martin’s Press. pp. 461-476. Nelson, B. J. (1984). Making an Issue of Child Abuse. Chicago, IL: University of Chicago Press. Page, B. I. (1996). Who Deliberates?: Mass Media in Modern Democracy. Chicago, IL: University of Chicago Press. Page, B. I. & Shapiro, R. Y. (1992). The Rational Public: Fifty Years of Trends in Americans Policy Preferences. Chicago, IL: University of Chicago Press.

14

Patton, M. Q., Grimes, P. S., Guthrie, K. M., Brennan, N. J., French, B. D., & Blyth, D. A. (1977). “In Search of Impact: An Analysis of the Utilization of Federal Health Evaluation Research,” in C. H. Weiss (ed.) Using Social Research in Public Policy Making, Lexington, MA: Lexington Books, pp. 141-164. Patton, M. Q. (1978). Utilization-Focused Evaluation (1st ed). Beverly Hills, CA: Sage Publications. Patton, M. Q. (1997). Utilization-Focused Evaluation: The New Century Text, (3rd ed.) Beverly Hills, CA: Sage Publications. Rog, D. J. (1985). A Methodological Analysis of Evaluability Assessment. Ph.D. Dissertation, Vanderbilt University, Nashville, TN. Rutman, L. (1988). Planning Useful Evaluations: Evaluability Assessment. Beverly Hills, CA: Sage Publications. Scarr, S. (1998). “American Child Care Today.” American Psychologist, pp. 95108. Scheirer, M.A. (ed.) (1996) “A User’s Guide to Program Templates: A New Tool for Evaluation Program Content.” New Directions for Evaluation, 72, San Francisco, CA: Jossey-Bass. Shulha, L. M. and Cousins, J. B. (1997). “Evaluation Use: Theory, Research, and Practice Since 1986.” Evaluation Practice, (18) 3, pp.195-208. Shulock, N. (1999). “The Paradox of Policy Analysis: If It Is Not Used, Why do We Produce So Much of It?” Journal of Policy Analysis & Management, 18 (2), pp. 226-244. Stimson, J. A. (1998). Public Opinion in America: Moods, Cycles, and Swings. (2nd) Boulder, CO: Westview Press. Stufflebeam, D. L. (in press, 2001). “Foundational Models for 21st Century Program Evaluation.” New Directions for Evaluation, 89, San Francisco, CA: Jossey-Bass. Weikhart, D. P. and Schweinhart, L. J. (1997). “The High/Scope Preschool Curriculum Comparison Study Through Age 23.” Early Childhood Research Quarterly, 12 (2), pp. 117-143. Weiss, C. H. (1983). “The Stakeholder Approach to Evaluation: Origins and Promise,” in Anthony S. Byrk (ed.), Stakeholder-Based Evaluation. New Directions for Program, no. 17., San Francisco, CA: Jossey-Bass. Weiss, C. H. and Bucuvalas, M. J. (1980). Social Science Research and Decision-Making. New York, NY: Columbia University Press. Weiss, C. H. and Bucuvalas, M. J. (1977). “The Challenge of Social Research to Decision Making,” in Using Social Research in Public Policy Making. Lexington, MA: Lexington Books, pp. 213-232. Wholey, J. (1994). “Assessing the Feasibility and Likely Usefulness of Evaluation,” in J. S. Wholey, H. P. Hatry, and K. E. Newcomer (editors) Handbook of Practical Program Evaluation. San Francisco, CA: JosseyBass Publishers, pp. 15-39. Zhu, J. H. (1992). “Issue Competition and Attention Distraction: A Zero-Sum Theory of Agenda-Setting.” Journalism Quarterly, 69 (4), pp. 825-836.

15