year Catchment Action Plan, developed with input from community and stakeholders ... A rolling three to five year Implementation Plan was created to bridge the ...
Creating Effective Use Of Evaluation Through An Operationally Embedded Adaptive Management Framework 1
Alexandra Murray ; Lyndal Hasselman 1 2
2
Grains Research & Development Corporation (GRDC), 40 Blackall St, Barton, ACT 2600 ANZSIG, University of Canberra, ACT 2601
Abstract Integrating learnings from monitoring and evaluation back into the planning cycle is not a new concept. However, it’s often implemented in a chunky, separate process and is likely to be undertaken intuitively or left as recommendations by the evaluation team. A new approach has been designed and embedded into the Lachlan CMA’s planning. The triple loop adaptive management framework questions across different levels of assumptions to elicit learnings and identify improvements. It clarifies roles across the organisation, empowers project managers to critically review, provides an opportunity for partners and stakeholders to share their values in the evaluation process and prompts strategic level questioning by the Board. Project managers are involved in the first level of questioning and adaptive management focusing around answering “Are we doing things right?”. Adaptive management at this level delivers refinements and improvements to create efficiencies and improve effectiveness toward the project objectives. The second level of questioning involves program planning and implementation. Here collective input on performance will enable refined implementation and management strategies and provide rigor in reporting to government and community on the extent of outcomes achieved answering “Are we doing the right things?”. Adaptive management will maintain, adjust or shift project objectives. The third level of planning involves strategic level catchment objectives and oversight to the decision making process by the Board. Explicit questioning of “How should we decide what’s right?” enforces planning, re-evaluation of values and implementation governance. The framework has clarified the role for evaluation in decision making and adaptive management for the effective achievement of both catchment and project level objectives. It provides a flexible and responsive approach to achieve the catchment plan’s objectives, making planning living and resilient. The framework has been published online as Support Chapter 7 Monitoring and Adapting the Catchment Action Plan (Lachlan CMA, 2013)
Background context The Lachlan Catchment Management Authority (Lachlan CMA) was established in 2004 in accordance with the NSW Catchment Management Act 2003. The organisation is overseen by a Board, as appointed by the Minister for Primary Industries and works to improve natural resources of the Lachlan Catchment by working with land managers, community and stakeholders. A review and upgrade of the Lachlan CMA’s guiding strategic document, the Ministerial approved Catchment Action Plan, created an opportunity to review and change the monitoring and evaluation frameworks employed by the Lachlan CMA.
Evaluations conducted at the Lachlan CMA have focused on whether change has occurred, the nature and degree of change, and the factors that lead to change. Assessing, understanding, and explaining change is at the center of the evaluations. However the evaluations tend to assume standardised, controlled, replicable, high fidelity interventions that can predictably produce standardised outcomes. The standard evaluation question has implied and assumed a static and fixed relationship between cause and effect, between program and outcome, between intervention and impact. Evaluation in complex social-ecological systems, with dynamical interactions and interdependencies rarely fits this approach to evaluation. The use and adoption of evaluation findings into subsequent project and program design is difficult to trace in many instances. Knowledge has been gained by evaluators and staff involved directly in conducting evaluations, however the sharing of this knowledge more broadly through the organisation is debatable.
Introduction to triple loop learning Triple-loop learning is a conceptual framework for organisational learning, stemming from the work of Argyris and Schön on double and single-loop learning. Single-loop learning occurs when deviations from goal are realigned without questioning the goal itself (Argyris, 1976). These are small incremental refinements required to maintain direction (Pahl-Wostl, 2009). It can occur through several iterative adjustments in adaptive management. The questioning and resulting adaptation does not challenge the end point goal, or how we decided on the end point goal, but seeks to refine how we are getting there. It questions ‘Are we doing things right?’, uses tighter, close feedback loops, and is efficient feedback to improve how we go about doing our business. Single-loop learning refers to the process of modifying actions as a result of an evaluation between the desired and actual outcomes, making incremental improvements and changes to achieve the desired outcomes. Double-loop learning questions the rules, guidelines, contexts, variables and goals of actions and as such, requires the capacity for critical self-examination (Pahl-Wostl, 2009; Argyris, 2002; Argyris, 1976). It questions end point goals and prompts management to reframe the question at the centre of evaluation to ‘Are we doing the right things?’. Feedback here informs about the approach or method used to reach the end goal and the end goal itself. In program logic approaches to evaluation, this is about checking both causality and assumptions with the theory of change and the longer term desired outcomes and objectives. The concept of double loop learning has several important implications for adaptive management by changing the emphasis in problem-framing. Redefining the questions and problems facing an organisation often reveals liabilities and shortcomings in organisational culture and structure that if not attended will leave that organisation at risk. In addition, feedback from the environment and feedback from the intervention or action are both important in double-loop learning, requiring those seeking to reflect on ‘are we doing the right things’ to also look externally for insights on impacts and contexts (Argyris, 1976; Tosey, et al., 2012). The third loop looks at how learning takes place and challenges beliefs and values that underpin the decision making process with questioning of how we decided what’s right, asking ‘How should we decide what’s right?’ Changes that arise from the triple-loop are transformative (Pahl-Wostl, 2009). This deliberative questioning probes our underlying deeper held assumptions on our values and beliefs that have contributed to decision making and choice selection. This form of learning helps us to understand more about our own and other’s beliefs and perceptions. It is through triple loop learning
that an individual or organisation can determine how they need to be different to create transformational and sustainable change. The results of this learning includes increasing ways of understanding and changing our purpose, developing better understanding of how to respond to our environment and deepening our comprehension of why we choose to do the things we do. The rationale and purpose of the organisation can come into question in triple-loop learning (Tosey, et al., 2012). The frame of triple loop learning is considered particularly useful in the governance of natural resource management as a means of learning and changing in response to changing contexts, the lack of repeatability in the ‘experiments’ of natural resource governance and the variability that is intrinsic to social-ecological systems and their resilience (Folke, et al., 2002; Garmestani & Benson, 2013; PahlWostl, 2009). Example 1. Impact of changing values on decision making. Modified from Support Chapter 7, Monitoring and Adapting the Catchment Action Plan (Lachlan CMA, 2013) Intensive hunting of Australian fauna from the mid-1800s until the 1930s occurred because they were regarded as either agricultural vermin or valued for their skins, meat and other by-products. The extent of the scale of hunting included 300,000 Koala skins sent to London in 1889 and a further 600,000 Koala skins purchased by the NSW Government in 1902. In 1935 every house along the banks of the Lachlan River was pegged out with Koala skins. At the single loop, European settlers could focus on “yes we are doing things right” because we successfully killed a lot of Koalas. In the 1930’s with increasing scarcity of Koalas questioning “are we doing the right things?” found that a change in resource choice towards fox and rabbit had comparative advantage. A change in society’s values also became more prevalent, in that the koala became an invaluable intrinsic resource. In terms of the original, now archaic decision for government purchase of Koala, it can be questioned how this decision came about, how it reflected the values of the time and if the process of decision making used was appropriate, and “how should we decide what’s right?”
The design of a triple loop adaptive management framework The planning and governance infrastructure at the Lachlan CMA is hierarchical. At the top is the tenyear Catchment Action Plan, developed with input from community and stakeholders and approved by the Minister for Primary Industries. Implementation of the Catchment Action Plan is overseen by the Board of the Lachlan CMA; its members directly appointed by the Minister on a skills basis. The Catchment Management Act 2003 also requires an Annual Implementation Program to be developed and submitted to the Minister for approval. This Annual Implementation Program demonstrates how state funding and resources will be used to implement the Catchment Action Plan. It provides the expected deliverables, in terms of outputs, for the financial year and is developed internally by the Lachlan CMA. Following the Annual Implementation Program, project plans are developed by project managers and teams and submitted to the Lachlan CMA’s Management Team. These project plans provide operational detail of how the Annual Implementation Program will be achieved with project level objectives and methods defined along with the resourcing allocated.
During the review and upgrade of the Lachlan Catchment Action Plan, a fourth level of planning was developed. A rolling three to five year Implementation Plan was created to bridge the temporal scales between the ten year Catchment Action Plan and the Annual Implementation Program and allow for community and stakeholder input into implementation level planning and decision making. In this newly designed level, a Partnership Planning Forum is held every year with local knowledge holders, technical experts and strategic partner organisations involved in natural resource management within the catchment. The Partnership Planning Forum is tasked with reviewing the collective results of the previous year and nominating appropriate interventions for the coming financial year. The results of the Partnership Planning Forum form a three to five year rolling Implementation Plan that is then used by all parties, including the Lachlan CMA, to plan their own individual annual investment portfolios, or in the case of the Lachlan CMA; the Annual Implementation Program that is then submitted to the Minister. The adaptive management framework presented here was designed to work with these levels of governance. There has been some simplification of the triple-loop learning process. This was made in acknowledgement of the shift the framework represents and the collective capacity required to fully and positively implement triple-loop learning. It was an expectation of the authors that gaining initial organisational experience and familiarity would over time lead to the experience and ability to further develop the framework and more fully apply triple-loop learning. The main simplification in the application of triple-loop learning is the assignment of loops to levels within the planning and decision making hierarchy, as shown in Figure 1. Single-loop learning has been assigned to the project planning and implementation cycle. Double-loop learning has been assigned to the work of the Partnership Planning Forum and consequent three to five year Implementation Plan. Triple-loop learning has been assigned to the Catchment Action Plan and the deliberations of the Lachlan CMA’s Board.
Figure 1. Schematic of triple loop learning assigned to planning levels, as reproduced from the Lachlan CMA Catchment Action Plan 2013-2023 (Lachlan CMA, 2013)
Single-loop learning at the project level Single-loop learning, the iterative adaptive management process, was assigned to project planning and implementation, and to be handled within the project planning and implementation processes. Some learning and adaptive management may occur on an annual basis and be documented within annual final project reports whilst other smaller continual refinements may occur on a continuous basis as represented by the smaller internal feedback loops on the inner loop shown in Figure 1 (Lachlan CMA, 2013). At this level, monitoring and evaluation is operationally focused on ‘Are we doing things right?’ or did we achieve what we set out to achieve? It requires good business practice of knowing what the organisation is expending resources on and reporting the outputs. Incremental adjustments or decisions not to make adjustments to the implementation of project goals are applicable, and performed with the aim to achieve the approved project objectives and deliver targeted outputs. This iterative learning cycle of monitoring, assessment and decision making based on a continuum of adjustments through learned successes and mistakes, guides project implementation within the
constraints of approved project plans, resourcing and agreed outputs. Triggers or thresholds need to be identified to support the recognition of when outputs are not indicative of project objectives. Critically, project managers are guided to also question the why, why not and how, underlying their results. The insights from this underlying questioning are important content that requires feed up to the double-loop. In the absence such insights gained through the implementation of projects, double-loop learning would be poorly informed. In addition, when it is identified that adjustments to the project are required that shift the direction of the project, changing its scope and end point, this triggers a push into a higher level of learning, planning and decision making. The key success attributes at the single-loop learning level are that the project has delivered its intended outputs and objectives and that information for feed up has been gathered and provided. In defining successful delivery of intended outputs and objectives, projects may identify specific success attributes, particularly those pertaining to technical quality parameters (Lachlan CMA, 2013). Double-loop learning to develop the Implementation Plan Double-loop learning, the deeper questioning of assumptions and values, questioning, learning and adapting that focuses on ‘Are we doing the right things?’, is applied to the new annual process of implementation planning by the Partnership Planning Forum for the coming three to five years. In the second learning loop the evaluation questions of appropriateness and efficiency come into play while questions on impact and effectiveness remain important in guiding judgements on the relative performance abilities of different approaches. Double-loop learning is expected by the authors to gain insights that generate significant improvements and developments in natural resource governance within the catchment. Monitoring and evaluation is focused on identifying the appropriate choice of change mechanisms (policy and planning) to facilitate achievement of outcomes, as defined within the Catchment Action Plan. This challenges for the refinement, shifting or developing of new approaches or to find the ‘right things’ to implement in order to achieve Catchment Action Plan stated goals. Adding rigour to this challenge is information gathered from community and stakeholders on acceptable levels of performance. Progress or lack of progress towards these performance measures can serve as triggers for review, questioning and adaptive management. Results of the online survey, in comparison to past performance has highlighted areas where doing the same things as done in the past, will not achieve ‘good’ performance. There is a need to shift away from business as usual and discover new approaches in order to perform and excel (Lachlan CMA, 2013). Information needs in order to judge if the right things are being done include project final reports and associated feed up of project level findings, any evaluations conducted into past interventions by any of the partners, context comparable evaluations and other new research insights into fostering change and improvement in natural resource management and information on changing community values or environmental variables as gathered through external scanning. In terms of project final reporting and evaluation, at this second learning loop information is required on the achievement of outcomes, both intended and unintended, rather than the single-loop learning focus on output and project objective. However, it is noted that the definition of the term ‘right’ used in this double-loop learning is value ridden. During the Catchment Action Plan development the Lachlan CMA’s Board developed a list of nine decision making guidelines. These guidelines, listed below in Example 2, form the basis of determining ‘right’ and are provided in the framework as the success attributes for the double-loop
learning level. The authors expect that deliberations of the Partnership Planning Forum will inadvertently test and challenge these guidelines and in doing so create feed up to the triple-loop learning level. Example 2. Lachlan CMA Board Decision Making Guide Lines (Lachlan CMA, 2013) • • • • • • • • •
Decisions and actions should align with broader Government directions for natural resource management and ecologically sustainable development. Investments and actions should work towards the vision for the future catchment and the visions for the future desired landscapes of the five social-ecological systems identified in this Catchment Action Plan. The outcomes of actions should increase the diversity and resilience of the social-ecological systems. Favour activities that maximise leverage, focusing on what can be influenced to make the most difference. Decisions must be substantiated with credible evidence and take into account community consultation. The social, economic and environmental contexts and implications of potential decisions should form a part of the decision making process. Favour social inclusion, motivate the community and others, encourage innovation and creative solutions. The results of decisions should provide clear direction and have measureable results and outcomes. Encourage shared responsibilities across whole of government and whole of community (Lachlan CMA, 2013)
Triple-loop learning at the Catchment Action Plan level The largest loop in Figure 1 represents the Catchment Action Plan loop, where monitoring and evaluation is about challenging assumptions and questioning values and their ongoing relevance at a deeper level. This questioning level can work to reduce the bias of individuals, resulting in a holistic community approach and promote alignment between the decisions of the organisation with the values of the catchment communities. This level seeks to learn how to learn through questioning, learning, and adapting focus on ‘How should we decide what’s right?’. Questioning in the third loop can lead to radical transformation of organisations and is considered within the Lachlan CMA Board’s role and responsibilities of providing strategic directions (Tosey, et al., 2012). A category of knowledge that the Catchment Action Plan level must also deal with is the expectation of transformational events. These transformational events occur through biological or social systems. Anticipated transformational events include climate change or legislation that will cause repercussions on the social-ecological systems in the Lachlan. This may occur and cause a situation where the stated desired outcomes in the Catchment Action Plan may not be achievable and the second learning loop fails to facilitate progress toward the achievement of outcomes through appropriate change mechanisms (policy and programs), feedback will then be elevated to this third learning loop. Here resolution is sought to determine whether a transformational change of beliefs or a shift in world views has occurred and prevented achievement of the Catchment Action Plan. The framework currently defines the success attributes of ‘how should we decide what’s right’ as: • •
Alignment of the Catchment Action Plan to the values of its communities, Alignment of the Catchment Action Plan to government policy (as a proxy for whole of community views), eg. alignment to the principles of ecologically sustainable development
•
including societal and intergenerational values, and Alignment of the process and plan to the ideals of good governance (delivering results, efficiency, probity, transparency, informed decision making) (Lachlan CMA, 2013).
Discussion on the adaptive management framework It is widely acknowledged that ecosystems are changing more rapidly and less predicably. Globally, governance of natural resource management has been criticised for its inability to work with or be flexible and responsive to ecosystem change due to command and control paradigms, legal and institutional rigidity and assumptions of stability and predictability of social-ecological systems (Folke, et al., 2002; Garmestani & Benson, 2013; Pahl-Wostl, 2009; Bodin & Crona, 2009; Walker, et al., 2002). The framework designed attempts to increase flexibility for adaptive management and seeks to improve the effective management of natural resources, for the ultimate aim of improving the resilience of social-ecological systems within the catchment. Three of the advantages expected to be gained through the design of the framework and its implementation process are briefly discussed. Delegation of adaptive management responsibilities The triple loops provide a framework for making decisions and creates a continuum of shared learning across levels within the organisation and externally with key stakeholders and partners. The structure provided by the three loops and their allocation to parts of the planning and implementation structure clarifies responsibilities. All parties involved in natural resource management governance in the catchment have some role in monitoring, evaluation and learning, with responsibilities differentiated on the basis of assigned learning loop. The clarification of responsibilities provides for flexibility and freedom for adaptive management across the government hierarchy of natural resource management. Project managers and project staff are responsible and have the flexibility to implement continual refinement and adaptation within the rules and structures of the organisation, as per single-loop learning. Project managers and project staff are also responsible for gathering information and insights pertaining to the underlying causes and contexts of achievement of planned project level objectives and deliverable outputs and for feeding this information up to the double-loop. Participants of the Partnership Planning Forum are responsible for questioning assumptions, causality and achievement of outcomes in addressing appropriateness of interventions and developing the partnered rolling three to five year implementation plans. The Partnership Planning Forum has the flexibility to adjust and change approaches used to achieve the Catchment Action Plan. The Board of the Lachlan CMA is responsible for deep questioning on values and beliefs that underlie decision making and rule formation and in ensuring alignment with the values of the community. The Board, with its responsibilities to the Minister for Primary Industries, is also considered the most appropriate to address any emerging misalignment between community values and the policies and objectives of the Government. The legislative responsibility of the Board and Minister to approve the Annual Implementation Program remains. In the framework adaptive management of the plan does not require Ministerial approval or endorsement. Strategic planning at an outcome level results in a need for Ministerial approval only if evaluation findings necessitate changes in the direction or end point, not if adaptive management changes how the end point is reached.
Through this clarification and separation of duties it is hoped that adaptive management proceeds without undue administrative or governance burden, resulting in improved levels of performance. It is possible that through empowering project managers and clarifying roles that a culture of receptivity may continue to grow, resulting in increased overall organisational learning (Argyris, 1976). Cross scale communication including participation by local knowledge holders The adaptive management framework includes feedback, feed up and increases communication across stakeholders in natural resource management within the Lachlan Catchment. Processes are defined for information sharing up and down the management hierarchy and across the landscapes within the catchment. This is a significant increase in communication and exchange of information between levels of the organisation, as required for organisational learning (Garmestani & Benson, 2013). Furthermore, the Partnership Planning Forum creates an event for significant information sharing with those external to the CMA and for those external to the CMA to provide their knowledge and experiences into evaluation and planning on a regular basis. This communication, exchange of ideas and sharing of perspectives across land managers, community members, local organisations, scientists, project managers and CMA management is known to contribute to innovation, provide quicker feedback on results and observations, enable more frequent testing of values and appropriateness and reduce risk of resistance during implementation (Pahl-Wostl, 2009; Bodin & Crona, 2009). The dialogue aspect of the learning framework is also considered critical in the development of organisational learning (Pahl-Wostl, 2009; Garmestani & Benson, 2013; Argyris, 2002). This requires adequate time for discussion within each annual cycle (Raymond, et al., 2010) and the forum event would need to span several days, demanding significant time commitment from the participants. It is possible that the inclusion of broader participation in decision making will change power structures (Pahl-Wostl, 2009). Newly created and existing social networks that the Partnership Planning Forum may foster or contain can have significant and differing effects on governance processes such as propensity for collaboration and reciprocity (Bodin & Crona, 2009). The evolution of a productive, collegiate network requires time (Bodin & Crona, 2009) and long term organisational commitment on behalf of the Lachlan CMA as the convenor of the forum. The inclusion of stakeholders into resource governance through the Partnership Planning Forum could be considered a step towards polycentric governance (Pahl-Wostl, 2009; Folke, et al., 2002). However in this regards, the Partnership Planning Forum and it implementation plan is acknowledged as creating an additional layer of governance with a poorly defined accountability (Pahl-Wostl, 2009). There is also potential for game play and politicisation of the process by forum participants, along with the potential for a loss of power held by the Lachlan CMA, and these possibilities would reinforce or stagnate learning to single-loop (Argyris, 1976). Ability to evaluate decision making The adaptive management framework, through articulating the values that should ideally underpin decision making in the set of Lachlan CMA Board’s decision making guidelines, opens up new possibilities for evaluation and accountability. The Partnership Planning Forum will, through discussion inadvertently identify any misalignments in values. Instances where decisions are made in contradiction or with a different interpretation of the guidelines are also expected to raise discussion.
In addition to this informal testing of the decision making guidelines, it will now be possible to formally evaluate the decisions made by the Board and the Lachlan CMA against these quality or value parameters. It will be discernible if the guidelines represent espoused theory or actioned theory. This unintended outcome of the adaptive management framework was not envisioned during its design but nevertheless holds significant potential for improving the processes and documentation of the decision making process, driving the organisation for high standards.
Conclusion The adaptive management framework demonstrates a significant shift in the approaches to natural resource governance, and is underway at the Lachlan CMA. The adoption of the framework shows courage in the leadership of the organisation and a willingness and interest in learning for the ultimate betterment of the catchment. The framework has clarified the role for evaluation in decision making and adaptive management for the effective achievement of both catchment and project level objectives. It provides a flexible and responsive approach to achieve the catchment plan’s objectives, making planning living and resilient. Full realisation of the benefits of the approach will take time to develop as the framework is tested through implementation. Time is required to gain experience in implementing the framework, build reflective capacity and shift organisational culture to a learning frame. Over time and with development of learning competence, the organisation and its partners may be able to more fully apply triple loop learning. This may involve expanding considerations beyond project planning and implementation to other aspects of the organisation, applying multiple loops to each level of planning and developing structures for greater vertical and horizontal communications between organisations. While unforseen changes in the government structures for natural resource management in NSW may prevent, hinder or postpone implementation of the framework, implementation of the framework has potential for significant learning in organisational development and natural resource governance. Its implementation requires support and documentation of the process itself to elicit further understanding and development of natural resource governance towards learning cultures of increasing effectiveness.
Bionote: Alexandra Murray was the Lachlan Catchment Management Authority’s Catchment Coordinator for Monitoring, Evaluation, Reporting and Planning. She was the lead author of this adaptive management framework in the Lachlan Catchment Action Plan 2013-2023. Alex has drawn on her experiences and knowledge in agricultural extension and evaluation of international disaster recovery projects to develop this adaptive management framework and integrate it into the operational context of delivering natural resource management interventions. Alex is now working in an industry research and development role at the Grains Research & Development Corporation (GRDC). Lyndal Hasselman was Acting Program Manager at Lachlan Catchment Management Authority, project managed the development of the plan, provided oversight and guidance to the development of the framework. Lyndal has now commenced a PhD looking at the application of systems thinking to rural governance at the University of Canberra.
Bibliography Argyris, C., 1976. Single-Loop and Double-Loop Models in Research on Decision Making. Administrative Science Quarterly, September, 21(3), pp. 363-375. Argyris, C., 2002. Double loop learning: teaching and results. Academy of Management Learning & Education, December.1(2). Bodin, O. & Crona, I. B., 2009. The role of social networks in natural resource governance: What relational patterns make a difference?. Global Environmental Change, Volume 19, p. 366–374. Folke, C. et al., 2002. Resilience and Sustainable Development: Building adapative capacity in a world of transformations, Stockholm: Environmental Advisory Council. Garmestani, A. S. & Benson, M. H., 2013. A framework for resilience-based governance of socialecological systems.. Ecology and Society, 18 (1)(9). Lachlan CMA, 2013. Catchment Action Plan 2013. [Online] Available at: http://www.lachlan.cma.nsw.gov.au/downloads/Catchment_Action_Plan/Lachlan_CAP2013_2023_Support_Chapters_web.pdf [Accessed 30 July 2013]. Pahl-Wostl, C., 2009. A conceptual framework for analysing adaptive capacity and multi-level. Global Environmental Change, Volume 19, pp. 354-365. Raymond, C. M. et al., 2010. Integratinglocalandscientific knowledgeforenvironmentalmanagement. Journal of Environmental Management, Volume 91, pp. 1766 - 1777. Tosey, P., Visser, M. & Saunders, M. N., 2012. The origins and conceptualizations of 'triple-loop' learning: A critical review. Management Learning, 43(3), p. 291–307. Walker, B. et al., 2002. Resilience management in social-ecological systems: a working hypothesis for a participatory approach. Conservation Ecology, 6(1)(14).
