Exploring Evidence-based Policy Making in ...

Exploring Evidence-based Policy Making in Canadian Agriculture

Predrag Rajsic (Corresponding author) Postdoctoral Fellow Dept. of Food, Agricultural and Resource Economics University of Guelph Guelph, ON Canada N1G 2W1 e-mail: [email protected] Phone: 519-824-4120 ext. 56684 Fax: 519-767-1510 and Glenn Fox Professor Dept. of Food, Agricultural and Resource Economics University of Guelph Guelph, ON Canada N1G 2W1 e-mail: [email protected] Phone: 519-824-4120 ext. 52768

May, 2015 Selected paper prepared for presentation at the 49th Annual Conference of the CEA Ryerson University, Toronto, Ontario Friday May 29 — Sunday May 31, 2015

Exploring Evidence-based Policy Making in Canadian Agriculture

Abstract Evidence-based policy making (EBPM) is often invoked, but rarely applied. The available literature identifies several potential reasons for the lack of application of EBPM, including disagreements about appropriate policy goals and about appropriate means for achieving those goals. We review and assess the state of this literature to evaluate the extent to which it diagnoses the problem. We apply the theory of non-market failure to the current versions of the EBPM theory. According to Wolf's (1979) theory of non-market failure, market failure is a necessary but not a sufficient condition for policy intervention. The sufficient condition is that the proposed policy interventions do not result in nonmarket failure. Both market failure and nonmarket failure need to be diagnosed using the best available evidence, but the current versions of EBPM theory do not consider these types of evidence. Rather, it is taken as given that some policy intervention is needed, and the task of EBPM is to select the best policy among the competing alternatives. We find that this process is missing at least two steps: (1) providing sufficient and convincing evidence that a market failure has occurred, and (2) providing sufficient convincing evidence that a nonmarket failure is unlikely to occur for a proposed set of policy interventions. We then discuss how these these two steps of providing evidence could be applied to Canadian agricultural policy. We also inform our theory by applying Fox's (2012) analysis of the five theories of property rights.

Keywords: evidence-based policy, agricultural policy, market failure, nonmarket failure

1

Introduction Evidence-based policy making (EBPM) is often invoked, but rarely applied. While the official regulatory narrative seems to imply that both the definition and the application of EBMP are settled issues, available research indicates that the application of EBMP is limited at best and that there are numerous and often conflicting definitions of what constitutes proper evidence. The available literature identifies several potential reasons for the lack of application of EBPM, including disagreements about appropriate policy goals and about appropriate means for achieving those goals. There are frequent references to the need for more evidence-based policy at the federal and provincial levels in Canada. To illustrate this, we will provide several examples, but this list is by no means exhaustive. Policy Horizons Canada (also referred to as Horizons) (2013), describes itself as an agency within the Government of Canada that “conducts strategic foresight on cross-cutting issues that informs public servants today about the possible public policy implications over the next 10-15 years.” In its description of the role of evidence-based policy, Horizons identifies a need for “the best available objective evidence [emphasis added] from research to identify and understand issues so that policies can be crafted by decision makers that will deliver desired outcomes effectively.” This evidence, according to Horizons, “improves policy development in many ways, including by: reducing uncertainty, increasing logical clarity and consistency, providing new perspectives and understandings of policy issues, providing increased accountability to the public, providing reliable facts and knowledge, and improving the quality, inclusiveness and constructiveness of public policy debate.” Agriculture and Agri-Food Canada (AAFC) (2013) states that “[t]he growing expectations around accountability and value for money as actualized through program review processes, auditor general reports, and

2

evidence-based policy [emphasis added] and program development have provided further pressure for objective evidence linking public investment to outcomes.” The Canadian Centre for Agri-Food Research in Health and Medicine (CCARM) is an agency within AAFC that researches the health-related effects of functional foods and nutraceuticals (FFN). AAFC (2014) states that the mandate of CCARM is “to develop reliable, scientific, evidence-based information [emphasis added] concerning FFN's.” Ontario Ministry of Finance Commission on the Reform of Ontario's Public Services (2012) refers to evidence-based policy in its report on the reform of employment and training programs and states that “a recurring theme in this report is the need to base policies and programs on a defensible evidence base [emphasis added].” While these public statements seem to suggest a high importance of evidence-based policy on the public agenda, the available literature suggests that the extent of the application evidence-based policy making in Canada is inconclusive or modest at best. In this paper, we first review and assess the state of this literature to evaluate the extent to which it diagnoses the problem. Then we apply the theory of non-market failure to the current versions of the EBPM theory. According to Wolf's (1979) theory of non-market failure, market failure is a necessary but not a sufficient condition for policy intervention. The sufficient condition is that the proposed policy interventions do not result in non-market failure. Both market failure and non-market failure need to be diagnosed using the best available evidence, but the current versions of EBPM theory do not consider these types of evidence. Rather, it is taken as given that some policy intervention is needed, and the task of EBPM is to select the best policy among the competing alternatives. We find that this process is missing at least two steps: (1) providing sufficient and convincing evidence that a market failure has occurred, and (2) providing sufficient convincing evidence that a non-market failure is unlikely to occur for a proposed set of policy interventions.

3

We then discuss how these two steps of providing evidence could be applied to Canadian agricultural policy. We use an example of Canadian dairy, poultry and egg supply management policies to outline the important questions, consistent with the market and nonmarket failure theory of evidence-based policy, that need to be addressed. We also inform our theory by applying Fox's (2012) analysis of the five theories of property rights.

The State of Application of EBPM in Canada Young’s (2013) compiled a collection of summaries of current research on the application of evidence-based policy making in Canada. In his introduction to this collection, Young outlines what he sees as the sources of disagreement in liberal democracies, and he groups those disagreements into two categories: (1) disagreements about the policy goals or ends or purposes, and (2) disagreements about the means for achieving the goals. Young argues that these disagreement cannot be eliminated and that they are an integral part of what he calls reasonable pluralism or reasonable disagreement in liberal democracies. Young argues that one of the implications of reasonable pluralism is the need for justifying policy choices, and that EBPM is part of this justification process. Young’s evaluation of the Canadian situation is that the progress toward the implementation of EBPM is limited at best. He first reviews definitions of EBPM and finds that the common feature of these definitions is the idea that evidence-based policy needs to be grounded in “best available” evidence. However, what constitutes best available evidence is, Young argues, not well defined. Young goes on to say that most definitions tend to invoke evidence from natural or behavioural sciences, while the gold standard of evidence seems to be defined by the methods of evidence-based medicine. However, Young argues that EBPM is not

4

simply an extension of evidence-based medicine. The key difference between evidence-based policy and evidence-based medicine that Young identifies is that evidence-based medicine targets individuals while evidence-based policy targets populations. He argues that, because the decision on a medical intervention is ultimately up to the individual who will be subjected to the intervention, there is no need for justification of that decision. Since policy interventions affect groups of people, there is a need for justification of a particular policy intervention. Disagreements about proper evidence and about quality of evidence are, in Young’s view, at heart of the limited implementation of EBPM. Furthermore, he finds that. for many policy issues, evidence has not yet been produced. As another impediment to the implementation of EBPM, Young notes that evidence is only one of the factors that policy-makers take into account, alongside with time constraints, public opinion, political strategy, and election campaigns. Young concludes that implementation of evidence-based policy making may also be impeded by the lack of capacity (i.e., resources, technical expertise) or, in federal states, by a lack of coordination or conflicting policy objectives across different levels of government. The first essay in Young’s collection, by Howlett and Craft (2013), provides an assessment of policy advisory systems in evidence-based policy in Canada. Howlett and Craft’s thesis is that the theory of policy advisory systems is based on an outdated model of demand for and supply of evidence, where the demand for evidence comes from various government agencies, while the supply of evidence is provided by sources internal or closely linked to the government. In this model, which Howlett and Craft call the locational model, the importance and influence of evidence on policy is primarily determined by how closely linked the source of evidence is to the demand for evidence. They identify a number mutually related alternative emerging models that account for the content of evidence and how this content influences the

5

propensity of regulatory agencies to use evidence. Also, these alternative models account for the fact that evidence and policy advice come from a diverse set of sources (i.e., think-tanks, NGOs, colleagues, friends and relatives, members of the public and political parties). The emerging models also take into account the “politicization of policy-making” (Howlett and Craft, 2013; p. 37). Related to the politicization of policy-making, Howlett and Craft identify two main types of policy advice content: long-term and short-term content. They argue that Canadian policy tends to be focused on short-term advice, which tends to reflect a politicized reaction to an immediate crisis. However, Howlett and Craft point out that the short-term type of advice is ill suited for addressing complex, long-term issues like health care or environmental issues. They conclude that policy advisory systems in many Canadian sectors may have low propensities and preferences for EBPM, and thus may not be conducive for the application of EBPM. The second essay in Young’s collection, by Levin (2013), explores the institutional infrastructure for knowledge mobilization and research use in education policy. To represent the components and the process of knowledge mobilization in education, Levin uses a conceptual framework with three partially overlapping and interacting segments: (1) production (of knowledge) by universities and others; (2) mediation of knowledge (by individuals and organizations); and (3) use of knowledge (by policy makers and practitioners). The main interactions that affect the way in which research knowledge is used and applied, Levin argues, include the personal experiences and relationships of individuals within the three segments, collegial knowledge and organizational cultures, and social pressures in the workplace. The implication that Levin draws from this is that research knowledge itself is a necessary but not sufficient condition for policy change. The sufficient condition is that there are favourable conditions for the use of research knowledge, but Levin argues that in many cases the conditions

6

are not favourable. For example, he provides evidence that “many research-producing organizations are not effective at communicating their findings or the implications of those findings” (Levin, 2013; p. 53) either due to a lack of incentives or due to resource constraints. At the same time, research users “do not read very much original research” (Levin, 2013; p. 55) due to a lack of time or skill to interpret and apply research in daily practice. Levin points to various studies that document that “school systems, as well as departments and ministries of education, are also quite weak in this regard” (Levin, 2013; p. 55). When it comes to intermediaries, who translate and transmit research knowledge to potential users, little is known about their nature and work, Levin notes. The role of media as knowledge intermediaries, including social networks like Facebook and Twitter, although it could be significant, is unclear at this point, Levin continues. One often overlooked potentially powerful source of knowledge mediation between researchers and practitioners, according to Levin, are graduate students employed as temporary practitioners. Levin (2013; p. 63) concludes that despite “much rhetoric on the importance of evidence and a reasonable degree of activity as well … much less has been done to build the take-up capacity of the education sector.” In the next essay, Cooper (2013) explores further the nature, structure and functioning of research-brokering organizations in education in Canada. Cooper first identifies examples of research brokering organizations, including ministry research branches, district-level research service teams, standards and evaluation organizations, national funding agencies, university research centres, advocacy groups, issue-based organizations that focus on one particular area, think-tanks, textbook publishers, instructional program vendors, research consulting companies, and various media outlets. Then, she outlines three main research-brokering activities: (1) producing research summaries; (2) knowledge mobilization events; and (3) and knowledge

7

mobilization networks and argues that producing research summaries is less effective in knowledge transfer compared to knowledge mobilization events and networks. Cooper also finds that most research brokering organizations, with the exception of think-tanks, predominantly engage in producing research summaries and posting them on their websites. When it comes to dissemination of research, research-brokering organizations use a variety of methods, including face-to-face interactions, media outlets, online platforms (websites, Facebook, Twitter, online forums, blogs, and YouTube channels). Cooper (2013; p. 89), however, had little to say about “what strategies and functions are most effective with different audiences in different contexts.” The remainder of the essays in Young’s collection provides five evaluations of the extent to which evidence-based policy making is used in various policy areas: early childhood education, crime prevention, poverty reduction, tax design, and environmental protection. White and Prentice (2013; p. 96) argue that there is vast evidence that early childhood intervention policies (i.e. public child-care or kindergarten programs) result in “higher IQ scores, better performance in school, higher high school completion rates, and, in later years, …higher incomes earned, fewer arrests, higher rates of both home ownership and ownership of a second car, lower use of welfare and other social assistance, longer marriages, and fewer births outside of marriage” for “vulnerable populations of children.” They use this to argue that the level of public provision of all-day kindergarten and child care in Canada is too low, which they mainly attribute to the “reversal from an evidence-driven path” (White and Prentice, 2013; p. 105) by the Harper government. Waller (2013) argues that crime prevention policies are more cost-effective ways of reducing crime rates compared to the traditional system of prosecution and punishment through the court system. Waller focuses specifically on programs aimed at young persistent offenders.

8

According to Waller (2013; p. 134), These programs would include enabling parents to “provide more consistent and caring family education to their children”, “enable victims to achieve a greater degree of satisfaction with the justice system”, “address bullying, dating violence, and peer violence.” Waller’s view is that Canada has made some progress toward the implementation of evidence-based policy, mainly through the creation of the National Crime Prevention Centre in 1998 and the Policy Centre for Victim Initiatives in 2001. However, Waller goes on to say that Canada “needs to develop a profession of preventive practitioners equivalent to the public health profession.” Laforest (2013) compares poverty prevention initiatives in Ontario and Quebec in the late 1990s and the 2000s. She argues that the nature of Quebec’s initiatives was less conducive to the application of evidence-based policy making compared to Ontario. The Quebec initiatives involved a wide range of non-governmental organizations, including feminists, religious groups, unions, anti-poverty associations, student organizations, co-operatives, local economic development groups, and popular education groups whose main political strength was in the widespread and loud public support, which eventually put pressure on the provincial government to translate most of the proposals put forth by these groups into policy. According to Laforest, this setting was not conducive to the implementation of evidence-based policy, whereas in Ontario, policy formation was top-down, with the government having consultations with selected stakeholders who needed to back their proposals by some evidence of potential effectiveness. This resulted in the public announcement by the Ontario government that it will reduce child poverty by 25% within five years on December 8, 2008. Laforest concludes that this case study illustrates the importance of social context in the unfolding and implementation of policy changes.

9

Phillips (2013) assesses the application of evidence-based policy making to the process of redesigning Canadian research and development tax credit policy for businesses. The application of evidence in this case had the purpose of determining whether the status quo system of promoting research and innovation through tax credits had a greater impact on innovation compared to an alternative system of direct subsides. Although research efforts, as Philips states, were extensive, the “critical finding was that available evidence was inadequate to permit rigorous, comparative performance evaluation of different programs or estimate reliably their economic benefit or cost” (Phillips, 2013; p. 184). However, Phillips still considers this case study as an example of a successful application of evidence-based policy in the sense that “decisions are not directly determined by clear and conclusive evidence, but rather the presentation of evidence creates pressures for a reframing of policy problems and potential solutions” (Phillips, 2013; p. 188). The last essay in Young’s collection by Winfield (2013) evaluates the application of evidence-based policy making in Canadian environmental policy. Winfield’s thesis is that the Harper government has undermined the application of evidence-based policy making in Canadian environmental policy by introducing the set of policies known under the banner Responsible Resource Development, which, according to Winfield, dramatically undermined the Environmental Assessment Act. The Responsible Resource Development policies were, according to Winfield, designed to circumvent the requirements for environmental assessment for most development projects involving natural resources. Winfield (2013; p. 199) goes on to say that the Harper government is “demonstrating unusual hostility to EBPM relative to ideological or political factors.”

10

Implicit in Winfield’s claim is that the arguments for conducting evidence-based policy making are inherently non-ideological. The other authors in Young’s collection (Prentice, 2013; Waller, 2013; Laforest, 2013) also tend to view evidence-based policy as an objective enterprise that could, but generally should not, be tainted by ideological considerations. In our view, the claim that ideology and evidence-based policy are separate and often conflicting issues introduces a misleading dichotomy that needs to be explored further. This exploration, presented in the next section, will reveal the normative dimension of evidence-based policy making.

Exploring the Normative Dimension of Evidence-based Policy Typically, the term ideology seems to have a negative meaning, so it may be useful to use a more neutral term—belief system—instead. We will argue that, since policy, including evidence-based policy, is inherently a normative enterprise, it is also inseparable from human beliefs about normative issues. For example, one of the arguments for why evidence-based policy should be implemented is that this form of policy making would provide greater overall benefits to society as a whole. Fox (2012) explains that this view is derived from the utilitarian moral philosophy, which is a core philosophy of the progressive movement. The view that good policies are those that increase overall social welfare will shape the types of policies that are proposed and the types of evidence needed to justify those policies. However, there may be other normative arguments for why policy based on best available evidence should be implemented. For example, Barnett (1992) argues that the purpose of public policies should be to protect individual property rights. From the property rights perspective, some policies that may increase total welfare might be deemed unjust and the evidence that supports those policies might be considered irrelevant. Fox (2012) recognizes at least five different, and often conflicting, normative theories used for

11

justification of public policies: classical-liberalism, utilitarianism, pragmatism, legal-positivism, and modern libertarianism. When applied to the same policy issue, the five normative theories often come to different conclusions as to whether a policy would be justified or not. Therefore, in the context of evaluating different policy options, we are not facing a choice between evidencebased policy on one side and ideology on the other, or some mix of evidence-based policy and ideology. Rather, we are facing a multitude of belief systems which then shape the scope and aim of evidence-based policy. These belief systems also shape the set of evidence that is considered valid or useful. Aside from implicitly asserting that meta-analysis and randomized controlled experiment are generally the most appropriate methods for policy evaluation and for collecting evidence, Young and others had little to say about the appropriateness of different research methods and policy evaluation criteria. Pawson (2009) and Cartwright and Hardie (2012), whose critiques we review and evaluate in the next section, are, both in their own ways, strong critics of metaanalysis and randomized controlled experiment as the gold standards for judging the effectiveness of policies. Pawson argues that policy, being an inherently social enterprise, needs to take into account the human context, which has no place in randomized controlled experiments or in meta-analyses. Cartwright and Hardie (2012) argue that it is hard to predict the effectiveness of policies in different contexts because it is impossible to control most of the factors that affect policy outcomes.

Evaluation of Research Methods in EBPM Cartwright and Hardie (2012) describe evidence-based policy as an alternative to policy by anecdote or magic bullet policy (assuming a singular mapping of policy cause to desired effect or

12

outcome). They explain (p. 73) that “the temptation [to assume magic bullets] is very powerful for politicians, the public, and the media.” Cartwright and Hardie maintain that reality is generally more complex than the magic bullet model of causality admits. In their view, evidence-based policy consists of two elements: Effectiveness and relevance. Effectiveness refers to policies that worked somewhere. Relevance refers to a policy that has a reasonable chance of being effective in the new context for which it is being considered. In the literature on evidence-based policy, randomized controlled trials are generally seen as the standard for determining effectiveness. However, Cartwright and Hardie maintain that randomized controlled trials offer evidence of effectiveness but do not address the requirement for relevance. More than randomized controlled trials are needed to really do evidence-based policy. The model of evidence-based policy described by Cartwright and Hardie could be represented as a multivariate regression model. This is not the way that they present or explain their model, but it would seem to be a reasonable interpretation of what they are saying, at least when viewed through and economic lens. Defining the relevant policy outcome for individual i as Yi, the model can be written as 𝐼

𝑌𝑖 = 𝛼𝑖 + ∑(𝛽𝑗 𝑥𝑖𝑗 ) + 𝜀𝑖 𝑗=1

where αi is an intercept term which is unique, potentially to individual i, xij is the value of one of the factors thought to influence outcome variable for individual Yi, one of which, say xoi, is the policy action, βi is the effect of marginal variations in xij on the outcome variable for individual i and εi is a random error term.

13

There is no necessary reason that the relationship needs to be specified as linear, however, nonlinear effects don’t figure prominently in Cartwright and Hardie’s analysis. For the most part, the policy variable, xoi, takes on the same value for all individuals in the model. One of the authors’ frequently used examples has to do with the effects of elementary school class sizes on academic achievement, as measured with reading scores. In this example, in the representation above, Yi would be the reading score for student i. Variations in the policy variable, class size, xoi, influence the reading score of student i through the coefficient βi. For example, suppose the policy under consideration is to decrease class size from 30 students to 25 students. The hypothesis behind this policy change is that Yi will increase for students in smaller classes. This implies that βo is negative. The value of the multi-variate regression interpretation of the Cartwright-Hardie model is that it becomes quickly clear that the reading score outcome for any particular student will also reflect the values of the other x variables, some of which may be unique to that student, and some of which might be common to sub-populations or subsamples of students. Consequently, the reading score outcome for a particular student may go up or down when class sizes are reduced, depending on variations in the values of those other x variables that occur contemporaneously with the change in class size. The regression model interpretation also makes us aware of long recognized problems like identification and endogeneity associated with empirical estimation of these types of relationships with historical social data. Pawson (2009) also recognizes the problems inherent in applying multivariate relationships to unique social contexts. As an alternative, he proposes an approach that he calls realism, which “provides the most comprehensive account of principles and practice, theory and method” as a foundation for the theory of evidence-based policy. By realism, Pawson (2009; p

14

21) means the application of the “generative model of causation” for explaining social phenomena, where generative model of causation looks “for causal powers within the objects or agents or structures under investigation.” According to Pawson, it is not empirical uniformity— the number of times we observe something happening—that convinces us of causality in social structures. Pawson uses this idea to critique the empiricist perspective on the “What works?” question. According to Pawson (2009; p. 21), what works is not discovered by “pooling data in search of programs with consistently powerful effects.” Rather, he argues that empirical observations should be used only as a guide for identification of causal relations, but not an ultimate test of causalities. To understand the causal effects, according to Pawson, social scientists need to use outcome patterns rather than outcome regularities when selecting optimal policies. Outcome patterns refer to all policy outcomes, successful, unsuccessful and partially successful, while outcome regularities refer to looking for policies that consistently produce desired outcomes. Pawson’s central idea is that the key determinant of whether a policy will give desired results is “if the subjects go along with the programme theory and choose to use the resources as intended” (Pawson, 2009; p. 24). Pawson argues that context affects policy outcomes by constraining the choices of stakeholders in a program, which then determines the three sources of evidence: (1) choice mechanisms (2); characterises and circumstances of subjects; and (3) patterns of impact. So, the question of what works should be rephrased to “what works for whom in what circumstances” (Pawson, 2009; p. 25). The lessons that Pawson draws form this is that implementation matters; the power of different stake holders needs to be taken into account; and institutional setting in which intervention is introduced needs to be taken into account.

15

The current approach in meta-analyses is to classify evidence in a hierarchy where some types of evidence (i.e., randomized controlled trials (RCT)) are valued more than others (i.e., professional and expert opinion). Pawson argues that this value scale is inappropriate and that it should be context-specific. Expert opinion, or even gossip, is, in Pawson’s view, sometimes as important as other forms of knowledge for successful program implementation. Pwason’s major objection here is that randomized controlled trials are not, in his view, the best method for inferring causal relationships in social settings. While randomized controlled trials in medicine are designed with the intention of removing the influence of human will (i.e., placebo effect) on the experimental outcomes, Pawson (2009; p. 52) stresses his earlier point that “human intentionality is not the confounding factor but the very medium through which such [policy] interventions work.” Therefore, the RCT is not appropriate for testing whether a policy intervention works or not, Pawson concludes. As a better alternative to meta-analysis of policy interventions, Pawson proposes a method that he calls a realist synthesis. This method is based on his view of policy mechanisms as highly context-dependent so that the factors like policy history, policy theory, mechanisms, and the reasoning of the stakeholders play an important role in shaping policy outcomes. As a consequence, Pawson’s realist synthesis does not give a categorical verdict on whether a program works or not. Rather, the aim of this method is to improve the theory on which a particular policy is based by synthesizing the available evidence on how the policy, or a set of conceptually related policies, worked or might have worked in different settings. Pawson applies his realist synthesis to three policies that differ in scope and complexity: (1) Megan’s Law, (2) youth mentoring, and (3) a set of policies that he calls “naming and shaming” (Pawson, 2009; p. 151). For each of these cases, Pawson provides a compelling case

16

for his realist synthetic approach. For example, in assessing the effectiveness of Megan’s Law1, Pawson first lays out the official theory of how the law was intended to work. This theory is broken down in four steps. Each of the steps contains a number of hypotheses about how the law would be implemented in practice, what incentives for different stakeholders (i.e., the public, the sex offenders, the police departments) this implementation would produce, and how these incentives would lead to outcomes. Pawson then examines evidence, which includes legal documents, case studies, reports, and community briefings, to assess the hypotheses contained in the official theory of the case. Pawson finds that most of the hypotheses need to be either rejected or modified to more accurately reflect reality. For example, the intended outcome of releasing the identity of a registered sex offender to local residents was increased security in the area (i.e., reduced rates of sexual recidivism by registered offenders). However, the actual outcome was increased harassment of registered offenders by local residents and no change in the rates of sexual recidivism. Pawson’s take-home message is that, in order to be able to give reasonable predictions about the effectiveness of a given policy, we need to compare the intended outcomes of the policy with the actual intentions, capabilities, constraints, and motivations of the stakeholders.

The Theory of Market and Nonmarket Failure as an Evidence-based Policy Framework In our view, Pawson (2009) offers a valuable alternative that can address some of the important shortcomings of the standard theory of evidence-based policy making. His alternative provides a more insightful method for developing the understanding of the mechanisms through which policies lead to specific outcomes. We would call this aspect of Pawson’s analysis the functional

Megan’s Law was the popular name of the US sex offender and registration and community notification programs, introduced in 1996 after the rape and murder of Megan Kanka. 1

17

analysis of evidence-based policy. Functional analysis takes the intended policy goals as given and then determines whether a proposed or actual policy will lead to a given policy goal. While this method can help in developing the understanding of why certain policies lead to certain outcomes in certain situations, it does not offer a clear theory for evaluating the intended policy outcomes or for determining the kinds of evidence that would be needed for such an evaluation. In some cases, the desirability of outcomes can hardly be disputed. For example, in the case of Megan’s Law, it was assumed that a reduction in sex offender recidivism is a desirable goal. When the costs of achieving this goal are negligible, it is hard to dispute its desirability. However, the implementation of most policies requires significant resources, and if the cost of achieving a policy goal is prohibitively high, then that policy cannot be implemented. It is clear that there is a range of policy goals that fall somewhere in between having negligible implementation costs and having prohibitive implementation costs. While the standard theory of evidence based policy does not offer a framework for evaluation policy goals within this range, Wolf (1979) offers a broader economic framework in which the theory of evidence-based policy making could be nested as a special case. According to Wolf's theory of nonmarket failure, market failure is a necessary but not a sufficient condition for policy intervention. The sufficient condition is that the proposed policy interventions do not result in nonmarket failure. Along with the well-established sources of market failure—externalities and public goods; increasing returns; market imperfections; and distributional inequity—Wolf identifies four sources of nonmarket failure. The four sources of nonmarket failure, according to Wolf, are: (1) internalities and private goals; (2) redundant and rising costs; (3) derived externalities; and (4) distributional inequity.

18

Internalities and private goals refer to the incentives of the members of nonmarket organizations to incorporate their personal self-interest into the functioning of an organization. Wolf lists the incentives for budget growth and incentives for the adoption of newest technologies as examples of internalities. Since the output of nonmarket organizations is measured by how much money they spend on different programs, increasing the budget of a nonmarket organization would be considered desirable by its employees. This often leads to “pressures to spend rather than save resources” or to “to spend funds quickly and plainly” to justify further budget growth (Wolf, 1979; p. 119). Related to budget growth are also incentives to acquire the newest technology without considerations for the relative costs and benefits of the new technology. Wolf uses health care and the military as prime examples of the pursuit of technical excellence without regard for the resulting social costs and benefits. Redundant and rising costs refer to the tendency “for production to take place within production possibility frontiers and for cost functions to rise over time” due to the lack of the profit and loss pressures (Wolf, 1979; p. 124). As one of the major sources of redundant and rising costs, Wolf sees in incoherent agency goals (i.e., bringing all students’ reading scores up to the mean) or gaols for which no known technology is available (i.e., providing “dignified” work for everyone). Derived externalities are the unintended consequences of actions of nonmarket organizations. For example, regulations that limit permissible profits, which are generally calculated on the basis return on capital, may cause inefficient substitution of capital for labour as an unintended consequence. The last source of non-market failure, distributional inequity, refers to the political decision-making process, which is by its nature hierarchical, and therefore gives more decision-making power to some people relative to others. This unequal distribution of

19

power, Wolf argues, creates opportunities for inequity and abuse (i.e., government contract obtained through bribery or discretionary favours to some groups and neglect of other groups). As a result of potential nonmarket failure, a policy intervention might lead to outcomes that are inferior compared to the putative market failure that the policy was initially intended to fix. These unintended consequences, Wolf argues, can be reduced by applying what he calls the implementation analysis. The implementation analysis aims at identifying potential sources of market and nonmarket failures at different stages of policy implementation. The application of the market and nonmarket failure theory to evidence-based policy making would have important implications. First, before any policy interventions are even considered or proposed, there needs to be credible evidence that one of the types of market failures has occurred. Second, even if there is credible evidence that market failure has occurred, this is only the necessary, but not the sufficient condition for policy intervention. The sufficient condition for policy intervention is that there is enough evidence to suggest that a potential nonmarket failure would not be worse than the market failure that the policy was intended to address. Ellig et al. (2013) provide an example of how Wolf’s theoretical framework could be applied in practice.2 They use a set of criteria that they call the regulatory scorecard to measure the use of evidence of market and nonmarket failure in the U.S. policy making. Their purpose is to determine the extent to which various policies proposed by US executive branch agencies in 2008, 2009, and 2010 have been accompanied with research aimed at identifying necessary and sufficient conditions for a policy intervention. Ellig et al. found that “that the quality of regulatory analysis is generally low, but varies widely.” In our view, applying the regulatory

Although Ellig et al. (2013) don’t explicitly refer to Wolf (1979), their method is consistent with the core theoretical principles on Wolf’s policy implementation analysis. 2

20

scorecard approach to Canadian policy, including agricultural policy, could offer a transparent practical evaluation of the extent to which evidence-based policy making is implemented in Canada. This practical evaluation would be theoretically grounded in the theory of market and nonmarket failure.

Application of Market and Nonmarket Failure Theory of Evidence-based Policy Making: The Case of Supply Management In this section we provide a preliminary exploration of the application of market and nonmarket failure (MNF) theory of evidence-based policy making using the example of Canadian supply managed industries. The purpose is to outline the basic steps of MNF theory of evidence-based policy making. Production and marketing of milk, eggs and poultry in Canada are regulated under a policy framework known as supply management. This framework sets farm level prices with a formula and allocates farm level production and distribution to processors through a quota system. Formula prices are supported by quantitative limits on imports as well as tariffs. Changes in domestic market demand conditions are accommodated through adjustments in the total amount of quota available. The origins of supply management can be found in the formation of the Canadian Dairy Farmers’ Federation in 1934, later renamed into Dairy Farmers of Canada in 1942 (Dairy Farmers of Canada (DFC), 2015). Dairy Farmers of Canada (2015) state that “the mandate of DFC evolved to pursue market stability policies and ensure fairer prices for producers.” According to the Canadian Dairy Commission (2011), supply management in primary milk

21

production was instituted “to address the unstable markets, uncertain supplies and highly variable producer and processor revenues that were common in the 1950s and 1960s.” When it comes to the production and marketing of milk, the Farm Products Agencies Act (2011 [1985]) established the legal basis for the operation of the Canadian Dairy Commission (CDC), a federal agency that governs the operation of milk supply management. The body responsible for estimating quota allocation among provinces is the Canadian Milk Supply Management Committee (CMSMC). Each province has a team representing its interests in the CMSMC. The decisions are made unanimously, where each province gets one vote. Section 23 of the Farm Products Agencies Act (2011 [1985]) states that “in allocating additional quotas for anticipated growth of market demand, an agency shall consider the principle of comparative advantage of production.” However, the Act does not refer to a particular definition of comparative advantage or details on the methods of observation or procedure by which the principle would be considered. This section of the Act seems to have been ignored up until the recent research focusing on the egg industry. The initial quota allocations were determined based on the historical production volumes. Danielle Goldfarb (2009) describes the current system for allocating new quota across provinces as complex and not readily described by a formula. Originally, allocation of provincial shares of quota was done on the basis of historical production shares before the implementation of supply management. Now, “The main allocation factor is 90 percent population, and 10 percent historical production” (Goldfarb, 2009, p. 19), which she indicates does not correspond to allocation on the basis of comparative advantage. Although the history of supply management in Canada is well documented, it is not clear to what extent the principles of what is now known as evidence-based policy making were

22

applied in the formation of Canadian supply management policy. Also, taking into account the current debate about the application of comparative advantage in the allocation of provincial production quota within supply managed industries, it would be useful to explore how evidencebased policy making could be applied in this case. The key questions that this exploration would need to address include: 1. Was there any evidence of o externalities and public goods; o increasing returns; o market imperfections; or o distributional inequities in the Canadian dairy, poultry and egg industries, and to what extent was this evidence considered during the policy making process? In case that this research reveals that there was sufficient evidence to suggest the existence of market failure, the next question that would need to be addressed is: 2. Was there any evidence of o internalities and private goals; o redundant and rising costs; o derived externalities; or o distributional inequities in the Canadian dairy, poultry and egg industries, and to what extent was this evidence considered during the policy making process? This approach would be more consistent with the general spirit of evidence-based policy making and it would produce a regulatory scorecard that could be used to inform future policy decisions.

23

Summary and Conclusions The purpose of this paper was to explore the implications of evidence-based policy making for Canadian policy, with particular reference to Canadian agricultural policy. To achieve this purpose, we first assessed the current state of the theory and of the application of evidence-based policy making in Canada. The available literature suggests that the theory of evidence-based policy making is somewhat incomplete and that its application is limited at best. There seem to be several potential reasons for the lack of application of EBPM, including disagreements about appropriate policy goals and about appropriate means for achieving those goals. We apply the theory of market and nonmarket failure to the current versions of the EBPM theory. In the context of evidence-based policy, both market failure and nonmarket failure need to be diagnosed using the best available evidence, but the current versions of EBPM theory do not consider these types of evidence. Rather, it is taken as given that some policy intervention is needed, and the task of EBPM is to select the best policy among the competing alternatives. The implication is that this process is missing at least two steps: (1) providing sufficient and convincing evidence that a market failure has occurred, and (2) providing sufficient convincing evidence that a nonmarket failure is unlikely to occur for a proposed set of policy interventions. The extent to which these two policy development steps have been used in Canadian agricultural policy is not well understood. We used an example of Canadian dairy, poultry and egg supply management policies to outline the important questions, consistent with the market and nonmarket failure theory of evidence-based policy, that need to be addressed. While further research is needed to obtain more definitive answers to those questions, this paper provides a more complete theoretical basis for that research, and it also informs our theory by applying Fox's (2012) analysis of the five theories of property rights.

24

References Agriculture and Agri-Food Canada. 2013. Evaluation of Performance Measurement and Reporting Programs – NAHARP and NCGAVS. Available: http://www.agr.gc.ca/eng/about-us/offices-and-locations/office-of-audit-andevaluation/audit-and-evaluation-reports/agriculture-and-agri-food-canada-evaluationreports/evaluation-of-performance-measurement-and-reporting-programs-naharp-andncgavs/?id=1379353990720 (Accessed April 2015). Agriculture and Agri-Food Canada. 2014. Cereal Research Centre. Available: http://www.agr.gc.ca/eng/science-and-innovation/research-centres/manitoba/cerealresearch-centre/?id=1180643854086 (Accessed April 2015). Barnett, R. 1992. “The Function of Several Property and Freedom of Contract” in E. F. Paul, F. Miller and J. Paul (eds.), Economic Rights. Cambridge: Cambridge University Press. Canadian Dairy Commission Online. 2011. Available: http://web.archive.org/web/20110210202207/http://www.cdc.ca/cdc/indexeng.php?link=114 (Accessed April 2015). Cartwright, N. and J. Hardie. 2012. Evidence-Based Policy A Practical Guide to Doing It Better. New York: Oxford University Press. Cooper, A. 2013. “Research Brokering Organizations in Education across Canada: A Response to Evidence-based Policy Making and Practice Initiatives” in S. Young (ed.), Evidencebased Policy Making in Canada. Toronto: Oxford University Press. Dairy Farmers of Canada. 2015. Our History. Available: http://www.dairyfarmers.ca/who-weare/our-history (Accessed April 2015). Ellig J, P. A. McLaughlin and J. F. Morrall. 2013. Continuity, Change, and Priorities: The Quality and Use of Regulatory Analysis across US Administrations. Regulation & Governance (7): 153-173. Farm Products Agencies Act (R.S., 1985, c. F-4 ). 2011. Department of Justice Canada. Available: http://laws.justice.gc.ca/en/F-4/index.html, (Accessed April, 2015). Fox, G. 2012. The Origins, Nature, and Content of the Right to Property: Five Economic Solitudes” Canadian Journal of Agricultural Economics 60(1): 11-32. Goldfarb, D. 2009. Making Milk: The Practices, Players and Pressures behind Supply Management, Conference Board of Canada, Ottawa. Howlett, M. and J. Craft. 2013. “Policy Advisory Systems and Evidence-Based Policy” in S. Young (ed.), Evidence Based Policy Making in Canada. Toronto: Oxford University Press.

25

Laforest, R. 2013. “Fighting Poverty Provincial Style” in S. Young (ed.), Evidence-based Policy Making in Canada. Toronto: Oxford University Press. Levin, B. 2013. “The Relationship between Knowledge Mobilization and Research Use” in S. Young (ed.), Evidence Based Policy Making in Canada. Toronto: Oxford University Press. Ontario Ministry of Finance Commission on the Reform of Ontario's Public Services. 2012. Available: http://www.fin.gov.on.ca/en/reformcommission/ (Accessed April 2015). Pawson, R. 2009. Evidence-based Policy: A Realist Perspective. Thousand Oaks, CA: SAGE Publications Ltd. Phillips, L. 2013. “Bringing Evidence to Tax Expenditure Design: Lesson’s from Canada’s Innovation Policy Review 2006-12” in S. Young (ed.), Evidence-based Policy Making in Canada. Toronto: Oxford University Press. Policy Horizons Canada. 2013. The Case for Evidence-Based Policy. Available: http://www.horizons.gc.ca/eng/content/case-evidence-based-policy (Accessed April 2015). Prentice, S. and L. White. 2013. “When the Evidence Doesn't Matter: Evidence-Based Policymaking and Early Childhood Education and Care in Canada” in S. Young (ed.), Evidence-based Policy Making in Canada. Toronto: Oxford University Press. Wolf, C. 1979. A Theory of Non-Market Failure: Framework for Implementation Analysis Journal of Law and Economics, 21(1):107-139. Waller. I. 2013. “Implementing Evidence-Based Policy to Deal with Crime in Canada” in S. Young (ed.), Evidence-based Policy Making in Canada. Toronto: Oxford University Press. Winfield, M. 2103. “The Environment, "Responsible Resource Development," and EvidenceBased Policymaking in Canada” in S. Young (ed.), Evidence-based Policy Making in Canada. Toronto: Oxford University Press. Young, S. (ed.) 2013. Evidence-Based Policy-Making in Canada. Toronto: Oxford University Press Canada.

26