How to Measure Capacity Development? A New Tool ...

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

How to Measure Capacity Development? A New Tool for the Evaluation of Intangible Assets in Regional Development st

1 October 2014 Silke N. Haarich, Independent Evaluator, Datteln, Germany

The Author: Silke N. Haarich holds a PhD in Economics and a Master’s Degree in Spatial Planning. She specialises in strategy design and evaluation of regional development strategies and programmes, especially in the field of cohesion policy and innovation strategies. She has wide professional experience in Europe and Latin America.

Contact: [email protected] Skype: silke_haarich Linked In/silke haarich

WORK IN PROGRESS PLEASE DO NOT QUOTE OR DISTRIBUTE WITHOUT PERMISSION OF THE AUTHOR

Abstract: Regional development is increasingly subject to evaluation in order to present specific results and socioeconomic impact. There is still a challenge if we refer to intangible assets needed to support regional development and innovation, such as networking, cooperation. This paper presents a new method to measure progress on intangible assets in regional development.

Key Words: Intangible Assets, Capacity Development, Evaluation, Regional Development, Methodology

1

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

1. Introduction Traditionally, economic development was based on three factors of production: land, labour and capital. During the last centuries and decades, more factors such as technology and knowledge have become decisive. Today it can be observed that “the traditional factors of production - land, labour and capital have not disappeared. But they have become secondary. They can be obtained, and obtained easily, provided there is knowledge” as envisioned by Peter Drucker (1993:42). The new relevant factor becomes visible in the buzz words that describe the keys of economic development in recent years: information society, knowledge economy, network society, Internet/digital economy. The ever-growing importance of the Information and Communication Technologies (ICT) and its influence on practically all fields of life and business is clearly the most important factor in this evolution. However, even if it also influences regional development and relevant policy-making, this research paper does not focus on ICT but on another important matter of the intangibles. Since intangible factors such as knowledge, data or information are becoming valuable assets for economic development and innovation, it is often said the intangible assets are generators of sustainable competitive advantage (e.g. Roth and Popescu 2012). However, with more knowledge and more technological opportunities, there is also more demand for intangible capacities at individual, organizational and sectoral/ regional level to absorb, analyse and manage increasing amounts of information and to cope with new and more complex societal problems. Knowledge as such is the raw material but it needs additional intangible conditions and capacities to prosper, such as networking, leadership, trust, effective cooperation, individual and organizational capacities to learn, manage processes or communicate etc. (Suriñach/Moreno 2011). In addition, other social trends feed the need for more developed regional capacities: •

a growing democratization of policy-making (rapid dissemination of information, participation, bottom-up approaches, mobilization through social networks, quadruple helix etc.),

•

more knowledge on the impact (or lack of impact) of policies with a corresponding demand for change and for innovative and improved approaches,

•

demand for effectiveness and efficiency in times of financial and economic crisis and limited public funding,

•

awareness of more complex problems that require systemic (public + private + social) solutions (climate change, economic-financial crisis, sustainability, poverty, ageing society etc.),

•

new technical possibilities open new ways of participation, information, communication, cooperation that are also increasingly demanded by stakeholders.

All these factors increase the requirements towards policy-making in multi-level and multi-agent settings with the corresponding needs for communication, cooperation, and consensus-building. Along the need for technical skills and capacities in each policy field, there is a growing need for functional capacities, both at administrative and political level. In this sense, political capacity refers to the ability of the decision-makers to make useful and effective decisions (based on values such as consistency, credibility, anticipation, long-term view) in a context of limited resources. Administrative capacity, in turn, refers to the ability to efficiently manage processes, resources, agents and instruments in order to achieve the policy objectives (Lember et al. 2013). This research paper presents some of the most relevant policy capacities and intangible assets needed for effective regional development (chapter 2), introduces some of the existing approaches to assess and measure intangible assets, capacities as well as changes through capacity development (chapter 3), and it presents a new approach to take into account these intangible assets during the evaluation of regional development (chapter 4). Finally, chapter 5 summarizes conclusions and hints for future research. 2

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

2. Intangible assets in regional development The importance of intangible assets in economic development becomes apparent when comparing the process of economic development with a road trip by car. In this image, the car represents the infrastructures and hard assets necessary for development (firms, resources, business activity, knowledge, production units, etc.). Support policies might add some lacking element to the vehicle (tire, rear-view mirror, etc.). The fuel would be available through financial and other resources (private and public). However, to really undertake the journey (of economic development) you would need also a driver (leadership), driving capacities (policy capacities), knowledge of the destination (vision), the itinerary (action plan), a certain control that your are on the correct way (milestones), as well as confidence to be safe (trust). These elements correspond to the soft factors or intangible assets in economic development processes. In all economic development processes, there are hard factors as well as intangible social and organizational assets involved (see Grabow et al. 1995). Depending on the perspective, as general intangible assets we can identify production factors within the firm (human and organizational capital, intellectual assets, brand name, etc.) and outside it (local externalities, the legal and institutional framework, the education system, property rights protection, social capital, among others (see also Suriñach/Moreno 2011:166). As the four core areas of intangibles in regional development, the European IAREG project establishes: knowledge capital, human capital, social capital and entrepreneurship capital (Suriñach/Moreno 2011:167 and IAREG project 2010). By nature, the whole field of intangible assets in regional development is still very fuzzy and vague, although its importance has been widely acknowledged. The qualitative, dynamic and emergent 1 character of these assets within the different regional systems makes it difficult to objectively measure or asses them. “An extensive difficulty in defining and specifying meaningful and sound indicators for capacity development is also that performance measurement cannot adequately determine the complex and process-oriented aspects” (ADA 2011:20). Because of these uncertainties, there is a need to know more about the intangible assets in regional development. “The European Union urgently needs a unified system for collecting data on intangible assets, such as human, organisational and social capital” (IAREG Project 2010). Especially with regard to EU cohesion policy 2014-2020 programming and the place-based strategies for regional economic development – in particular, the smart specialization strategies at regional level, that have been developed in all European regions as a prerequisite to access European funding – the need to asses and develop also intangible regional assets (social capital, governance capacities, political capacities and administrative capacities) by targeted measures becomes obvious. The strategies consider more and more the active support and development of intangible assets and capacities at regional level. However, when it comes to measure (in order to establish a baseline) and assess the actual level of intangible assets, important questions arise: a) which are the relevant intangibles?, b) which representative indicators could be used?, c) how could they be measured?, d) how could the assessment of intangibles be compared to other regions (similar approach, other approaches)? With the idea to develop an assessment tool oriented towards the intangible assets that could be complimentary with the already existing monitoring and evaluation tool box for regional development processes in the EU Cohesion Policy context, the relevant intangible assets must be limited further: Therefore we consider two different perspectives: 1. The social capital perspective and 2. The capacity development perspective. 1. From the social capital perspective, there are three important intangible areas to be contemplated (see Woolcock/Narayan 2000): •

institutional assets (transaction costs, trust, institutional agreements),

•

network assets (bonds, weak ties, bridges, structural holes), as well as 3

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

•

synergy assets (work and communication flows, dynamics, embeddedness, governance).

The three areas are complementary and sometimes overlapping. In addition, to these assets, we have to take into account the different overlapping levels: there are trustful relations with family and friends, but not necessarily with the public sector, within key organisations such as development agencies, or within governance entities, such as clusters or innovation centres. So, the multi-level perspective with a focus on the organizational and systemic (regional and sectoral) levels is extremely important. The social capital view allows for combining the three main elements of intangible assets.

Figure 1. Intangible assets for regional development from the social capital perspective

S y s t e m i c fe a t u r e s : n e t w o r k s , b o n d s (s t r o n g t i e s ) a n d b r i d g e s (w e a k t i e s ), tru st, lo w tra n sa c tio n c o sts, c o n se n su s, c o m m o n v isio n , sy n e rg ie s, c a p a c i t i e s fo r m u l t i -l e v e l g o v e r n a n c e

C iv ic so cia l ca p ita l, so cia l co h e sio n , ca p a citie s in civ il so cie ty to p a r t i c i p a t e , c o -d e c i d e a n d e xe c u t e

G o v e rn m e n t so c ia l c a p ita l, in stitu tio n a l (a d m i n i s t r a t i v e a n d p o litica l) ca p a citie s to p l a n , d e c i d e a n d e xe c u t e

M o r e e ffi c i e n t a n d e ffe c t i v e su p p o rt p o lic ie s, m o re im p a c t

E co n o m ic a n d so cia l d e v e lo p m e n t

Source: Own elaboration

2. On the other hand, during the last decades international development organizations have increased their interest in capacity development as an intangible complement to technical cooperation and necessary condition for development (e.g. ADA 2010, CIDA 2000, EC 2005, UNDP 2008, UNDP 2010). Figure 2. From Technical Cooperation to Capacity Development

Source: UNDP 2008:23

4

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

More than just training for individuals, “capacity development is seen as a long-term effort that needs to be embedded in broader, endogenous change processes that are owned by those involved, that are context-specific and that are as much about changing values and mindsets through incentives, as they are about acquiring new skills and knowledge.” (UNDP 2008:23) Capacity Development “is fundamentally about change and transformation – individual, organizational, societal. While it is purposeful in its orientation, it is not likely to be linear, easy or free of conflict. It is also about values – ‘whose capacity’ is to be developed, for what purpose? Finally, CD relies, in large measure, on learning and adapting behaviour, which requires an openness to the lessons of experience, and the flexibility to modify approaches as required.” (CIDA 2000:2)

Figure 3. Capacity Development Conceptual Framework

Source: CIDA 2000:8

The United Nations Development Programme (UNDP) proposes a systemic approach (with individual level, organizational level and enabling environment) focussing on four core issues (institutional arrangements, leadership, knowledge and accountability) and three areas of output to be improved: performance (effectiveness, efficiency), stability (institutionalization, risk mitigation) and adaptability (investment for innovation and continuous improvement) (see UNDP 2010). Another dimension within the UNDP framework is made up of technical (associated with specific sectors or themes) and functional capacities. “Functional capacities are ‘cross-cutting’ capacities that are relevant across various levels and are not associated with one particular sector or theme. They are the management capacities needed to formulate, implement and review policies, strategies, programmes and projects. Since they focus on ‘getting things done’, they are of key importance for successful capacity development regardless of the situation.” (UNDP 2008:12).

5

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

Figure 4. Functional Capacities as identified by the UNDP Capacity to engage stakeholders

Capacity to assess a situation and define a vision and mandate Capacity to formulate policies and strategies

Capacity to budget, manage and implement

Capacity to evaluate

• • • • • • • • • • • • • • •

• • • • •

Identify, motivate and mobilize stakeholders; Create partnerships and networks; Promote engagement of civil society and the private sector; Manage large group processes and open dialogue; Mediate divergent interests; Establish collaborative mechanisms. Access, gather and disaggregate data and information; Analyse and synthesize data and information; Articulate capacity assets and needs; Translate information into a vision and/or a mandate. Explore different perspectives; Set objectives; Elaborate sectoral and cross-sectoral policies; Manage priority-setting mechanisms. Formulate, plan, manage and implement projects and programmes, including the capacity to prepare a budget and to estimate capacity development costs; Manage human and financial resources and procurement; Set indicators for monitoring and monitor progress. Measure results and collect feedback to adjust policies; Codify lessons and promote learning; Ensure accountability to all relevant stakeholders.

Source: Own elaboration based on UNDP 2008:12

In short, intangible assets as we understand them in this paper are, on the one hand, specific systemic conditions (trust, bonds, networks), which can be largely described as social capital and are apparently difficult to be influenced in the short term. On the other hand, intangible assets are also specific functional (political, administrative, governance) capacities which are necessary for effective and efficient policy-making and the achievement of development goals. But, how can these intangible assets be assessed or measured and, in case that support is given to develop them, how can any change be observed or any impact evaluated?

3. Measurement of capacities and intangible assets: review of existing approaches There are numerous approaches to measure capacities or intangible assets that show relevant insights for the work with intangibles. But, in order to be a useful tool within the context of regional development systems, especially in the context of European Cohesion Policy, some basic requirements should be fulfilled. One major requirement is that an assessment at system level requires a rather complex methodology in order to detect and assess flexible, dynamic and intangible elements and processes. However, the tool should also support political decision-making. Therefore, the complexity level has to be reduced. In addition, to increase its usability and practicality within wider governmental frameworks, comparability with a certain goal or standard should be possible. So, the qualitative approach should seek somehow standardisation and comparability over time or with other regional systems. 6

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

In general, a useful approach should comply with the following requirements: • The approach must be applicable not only at (formal) organizational level, but especially with regard to informal or ad-hoc structures (PPP, networks, regional councils, clusters) at higher systemic levels (sectoral or regional governance). • The approach has to be easy to adapt to each case (region, policy field, context) and ease to handle, delivering meaningful insights and policy recommendations just in time for reflection and decisionmaking. More than a tool to judge or control, it should be a support for learning, self-assessment and development. • On a case-sensitive base, indicators should not only pay attention to capacities as an end in itself – losing sight of the big goals – but rather to the expected outcomes and impacts of improved capacities (effective and efficient policies, effective and useful governance structures, innovativeness, adaptability, economic development goals, social development, sustainable development, etc.). • The model should allow not only to answer questions about the “what” and “how much”, but also on the “why” and “how” of capacity development itself, facilitating learning on the intangible assets and on capacity development. Therefore, there should be ideally a combination of qualitative and quantitative indicators, and involve evaluation and impact assessment at later stages. • The model should pay attention to the different starting points of regional capacities and intangible assets. Regions and their structures are embedded in cultural, historic and administrative contexts and the assessment and the definition of goals has to acknowledge this environment in a flexible manner.

Keeping in mind these requirements, a series of existing approaches to measure capacities or intangible assets has been analysed. The table below describes the most relevant approaches, pointing out specific factors and conditions that make them more or less adequate in the context of evaluation of regional development projects and programmes.

Only few approaches seem really applicable for the (continuous) assessment of intangibles within regional development systems. In this context, it is essential to focus on the more complex level of (regional or sectorial) capacities where different actors and levels are involved and emergent dynamics play a key role to build intangible assets that are necessary for complex regional development processes. Although interesting models have been developed in the context of international development aid, focussing especially on capacity development in less-developed communities and countries, the analysis shows that there is still a need for a new model to measure capacities and intangible assets for regional development at a systemic level, especially adapted to the context of European Cohesion Policy.

7

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

Figure 5. Review of approaches to measure capacities and intangible assets (Source: own elaboration) Approach/ Model Organizational Assessment

Description / Examples There are several approaches to measure capacities and their performance at organizational level. Most models dedicated to the development or improvement within organisations (especially focussing on firms and non-profit, third sector organisations) usually start with an assessment of the initial state-of-art, so that they involve specific assessment instruments. There is, for instance, the six-box model as a framework developed by the analyst Marvin Weisbord to assess the functioning of organizations (used in EC 2005:15, see picture).

Another example is the Universalia Institutional and Organizational Assessment Model (IOA Model) (used in BID-IDRC 2002). Other models are the Seven-S model or the Burke & Litwin Model. (See also the Reflect&Learn web-page for more detail).

Organizational Learning

For the assessment of organizations in the business sector, there are comparable approaches such as the balanced scorecard model, the use of key performance indicators and, to some degree, the quality management approach. Related to the former approach but with different origins is the model that promotes organizational learning. Described by the sentinel work of Argyris and Schön in 1978, the theoretical concept would later influence the model of “learning organisations”, i.e. organisations that are able to learn and adapt (in particular though Senge’s “The Fifth Discipline” (1990). His five disciplines are: Personal Mastery, Shared Vision, Mental Models, Team Learning, and Systems Thinking. This more practical approach has become especially interesting to improve the 8

Review of adequacy for measuring intangible assets in regional development There are similar approaches to identify the relevant “capacities” or intangible assets that are considered as necessary to improve performance at organizational level. The IOA Model, for instance, defines as organizational capacities: strategic leadership, human resources, financial management, processes, programme management, infrastructures and institutional relationships. For each area it defines 2-6 components and relevant criteria, which are assessed by questions to the staff of the organization. The advantage is that the method is applicable in the context of particular organisations, (regional government, regional agencies, new governance entities), as long as they are aware of their need for change and interested in improvements. However, one inconvenience of these models is that they are focused on specific organisations (“how to improve the performance of organizations?”). Although they usually have a look also into the environment, the models do not apply a more holistic and systemic perspective with regard to regional or sectoral development (“how to improve the regional development through improved performance of organizations?”). This leads to the problem that one important feature in systems (networks, flows, links) is neglected by these models. The approach is attractive, because it adopts a more systemic perspective, integrating the individual, the group and the organizational level. It also offers a relatively simple methodology to establish an assessment for different levels, dimensions and factors. It is interesting to see in this approach that the

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

Approach/ Model

Description / Examples quality and performance within public sector and non-profit organisations (e.g. public administration, health sector, education). Among the numerous practical cases where this approach has been used are Nemeth 1997 and Dibbon 1999. For instance, Dibbon (1999) defines for a School five dimension of organization learning capacity (Individual Learning, Group Learning, Knowledge Management, Learning Dimensions, Learning Levers), as well as for each dimension specific factors (e.g. 6 factors for learning levers: Leadership for Learning, Building a shared Vision, Taking a Systems Perspective, Building a flexible organizational structure, Creating a collaborative Culture, Providing Resources for Learning). He defines for all areas different stages of development (coping stage, emerging stage, developing stage and learning stage), being the goal to achieve for all factors stage 4. For the assessment he uses a survey sheet and asks the organization members to evaluate the different factors in each dimension through pre-formulated phrases. He later uses a scoring system with higher scores for more developed stages.

Social Capital

During the 1990s, the general concept was applied to the regional level creating the “Learning Region”, especially used in the context of innovation support and regional innovation systems (see Cedefop 2007). Here, the importance of intangible assets and capacities is pronounced as a major factor for learning. However, neither the theoretical basis nor the assessment tools were really transferred to the regional level, so that no formal/ standardized methodology or practical application could be identified. At least since Putnam (1993), social capital has become a relevant field of sociology with relevant impacts on socio-economic development. Several studies have attempted to measure and quantify social capital and its contribution to economic development. Numerous academic works present valuable approaches and methods on how to measure social capital, either through combining socioeconomic indicators to “social capital indices” or by establishing causal links between development and the dimensions of social capital (e.g. Sabatini 2007). However, its multidimensional character and the diversity of the concept have made it impossible to date to define one standardized method for measuring social capital (Woolcock and Narajan 2000). One interesting practical tool is the SOCAT tool edited by the World Bank. The SOCAT is an integrated quantitative/qualitative tool. The Social Capital 9

Review of adequacy for measuring intangible assets in regional development development of capacities is not seen as a linear process (from 0 to 100), but as a dynamic process with several feedback loops (e.g. between individual and groups learning, or between learning levers and knowledge management). However, this means that the assessment becomes more complex. On the other hand, the systemic perspective makes it more adequate to analyse intangible assets which act at a systemic level (within organizations, or policy fields, or at regional or sectoral level). The method presented by Dibbon (1999) could be an interesting design to be adapted to a regional level in a particular policy context.

Social capital is one important element within the intangible assets for regional development. In the SOCAT tool, the following dimensions are analysed: Groups and Networks, Trust, Collective Action, Social Inclusion, Information and Communication. These are all important dimensions, also for more complex policies and development goals. The SOCAT tool can link social capital to development levels (poverty levels). However, the SOCAT tool lacks the possibility to compare the social capital of one community to an overall standard or general “stages of development”. This is understandable since the SOCAT instrument does not have the purpose to compare

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

Approach/ Model

Description / Examples Assessment Tool is a survey that measures individuals and groups' opportunities and constraints by focusing on the social assets and networks that determine their access to resources. (The World Bank, no year). It looks at several dimensions of social capital (Grootaert et al. 2002:4).

Social Networks

Capacity Assessment

Networks are actually an important part of the Social Capital (Burt 1997). However, since it is an important discipline in itself with its own methods (Social Network Analysis) it deserves a specific examination. The Social Network Analysis (SNA) is a method that allows to analyse the quantity and, to a certain degree, quality of networks and relationships between elements (persons, firms, organizations, disciplines, etc.) (see Borgatti et al. 2013). This is relevant, since most approaches focus on organizations or individuals (nodes) rather than on relationships (ties or flows). However, both objects of analysis are important when it comes to identify and measure systemic features and capacities. Still not used regularly in the context of regional development, SNA offers new opportunities to map and assess networking in specific regional contexts. Some examples are included in Giuffre (2013). The already presented UNDP approach focuses on the measurement of capacity development, especially related to capacity development programmes within international development (UNDP 2008 and 2010). As presented earlier in this paper, it parts from the hypothesis that capacities are important to create sustainable and demand-oriented development. Therefore, it defines core issues (institutional arrangements, leadership, knowledge and 10

Review of adequacy for measuring intangible assets in regional development different communities. However, guidance on the goals to be achievable and the next milestones for each dimension could become incentives for the development of social capital and specific support measures. SOCAT is a very comprehensive tool that enables to investigate in detail the development of social capital and the capacities of local institutions in less developed communities. Overall, it is an interesting tool to analyse a wide field of intangible assets for regional development. However, to be used in the context of European Cohesion Policy (in more developed communities and specific policy fields rather than poverty reduction), its methodology has to be adapted to the European context. For a regular use as a management instrument (among many others) it might be too comprehensive.

SNA is a complex method. Although supported by specific SNA software, it implies an important workload related to data collection and codification. Like in other statistical methods, analysis of the results is not straightforward but requires careful interpretation. This makes it improbable that SNA would be regularly used as a tool for evaluation in European Cohesion Policy, although it might support existing methods in some specific fields (regional innovation systems, interregional cooperation). The UNDP Capacity Assessment Framework is an interesting approach. It offers a general framework, flexible enough to be adapted to different regional and sectorial contexts. Additionally, it offers illustrative indicators (output, outcome and impact) for some fields of intervention. The model works like a toolbox and invites to define for

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

Approach/ Model

Description / Examples accountability) and three areas of output to be improved: performance (effectiveness, efficiency), stability (institutionalization, risk mitigation) and adaptability (investment for innovation and continuous improvement). The model invites to define a specific intervention path, based on inputs, desired outputs, outcomes and impacts for each field of intervention. It reminds also to see the capacity development along this path as a process in time (with evolving goals and indicators) that affects different levels (organizational, sectorial, regional, network level) (see UNDP 2010:5)

INTERREG Social Capital

INTERREG is the European Initiative to promote inter-regional cooperation among European partners. These projects usually have two sets of objectives: a) process-oriented goals to create or strengthen a partnership or specific types of cooperation and b) sector-specific goals. Although there is a focus on networking and other functional capacities, the diversity of projects and partnerships (many starting cooperation from zero), make it difficult to monitor outcomes and impacts at the level of cooperation, networking and related capacities. In 2006, INTERACT (FIORINI et al., 2006) proposed a new approach aiming at capturing the quality of partnerships and learning processes in INTERREG projects. The method is based on the Social Network Analysis (SNA) and the measurement focusses on: structural properties of partnerships, partnership dynamics (relational competency, norms and rules), as well as the external context in which the partnership operates (structures and institutional arrangements). The approach still has not been put into practice at a wider scale.

11

Review of adequacy for measuring intangible assets in regional development each case a specific intervention theory and narrative, establishing the desirable goals and milestones. However, this more open-ended, general approach requires experience, sensitivity for the case at hand and profound knowledge of capacity development processes and suitable indicators. While the definition of the specific intervention framework is an opportunity itself to develop capacities (for planning and monitoring) within local institutions, it might also become an obstacle for using this “advanced” framework, preferring an easy-to-transfer blueprint or existing model. Overall, it seems to be recommendable instrument to be used in capacity development programmes and evaluations in European Cohesion Policy. The fact that so many years have passed without finding a useful approach to monitor and assess partnership and cooperation-related goals and capacities of INTERREG projects, indicates the complexity of the matter. The approach presented here integrates very well the opportunities offered by SNA for the evaluation of quantity and quality of partnership networks in INTERREG cooperation projects. However, keeping in mind the complexity of the SNA method and the wide diversity of INTERREG projects it seems unlikely that standard indicators could be defined that are, on the one hand, easy enough to be used ty the project partnerships themselves, and on the other hand, meaningful enough to really depict the quality of cooperation. In addition, the approach lacks the necessary linkage between the intangible assets (cooperation, learning) to the final sector-specific goal. A partnership can be highly developed, but if its work does not contribute to the overall development goal, it is more meaningful to know the reasons behind this and not the details of the networking.

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

Approach/ Model Complexity Science

Collective impact (Developmental evaluation)

Description / Examples There are several approaches to analyse complex systems and to measure complex structures. These methods are more and more applied to field of development. Examples could be regional changes in climate and vulnerability to natural hazards, regional innovation systems, complex governance systems, etc. Methods used in this context are, the aforementioned SNA (Social Network Analysis), Agent-based or Multi-agent Models, Data Mining, Scenario Modelling, Dynamical System Modelling, etc. (OECD 2009).

A rather new approach is the “Collective Impact” model presented by FSG, especially but not exclusively for the non-profit, community development sector. “Because most shared measurement systems focus primarily on tracking longitudinal quantitative indicators of success, however, the systems are not typically designed to capture emergent dynamics within the collective impact effort—dynamics which are multi-dimensional and change in real time.” (Kania and Kramer 2011). It refers especially to the systemic level (inter-organizational), which makes it rather innovative. Also, it tries to identify and improve certain key dimensions (capacities or intangible assets) to be developed by various actors. A recent guide presents the model that links on-going assessment with (developmental) evaluation. (picture in Preskill et al. 2014:12).

12

Review of adequacy for measuring intangible assets in regional development These methods are based on complex scientific research, requiring time-consuming field work (or modelling) and advanced knowledge of specific IT programmes, which makes them less useful for policy evaluation. In general, methods can become really complex if they want to reflect the systemic dynamics in regional development, because they have to measure processes rather than entities, capabilities (potential action) rather than real action. In the context of assessing intangible assets, models are especially useful, e.g. to visualize multi-agent activities or to understand complex phenomena such as climate change impact at local scale. This new approach links the more traditional and intraorganizational focussed approach to assess Organizations (first row in this table) with the more flexible approach to assess evolving capacities and dynamic properties of larger systems (UNDP 2010). Like the UNDP Capacity Assessment model, this approach considers the evolution in time and the different needs of rather new or more developed institutions and systems. The five key dimensions or capacities that should facilitate improved performance and the multiplying of impact are: common agenda, backbone infrastructure, mutually reinforcing activities, shared measurement and continuous communication. The model defines sample outcomes and sample indicators for different cases and intervention fields. This model presents also guidance for the evaluation of capacity development initiatives. In general, it gives adequate hints to assess intangible assets for regional development at a systemic level. It has to be adapted, however, to the context of European Cohesion Policy and to each specific case.

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

4. Proposal of a model to measure intangible assets for regional development The brief analysis in the previous chapter demonstrates that there are several approaches to assess the capacity and intangible assets within organizations, but only few approaches that focus on the more complex level of (regional or sectorial) systems where different actors and levels are involved and emergent dynamics play a key role to build intangible assets that are necessary for complex regional development processes. The challenge is to find a methodological approach to measure intangible assets that do not ignore the complex environment and the systems dynamics of regional development processes, but are simple and handy enough to permit a use in evaluation and a rapid and meaningful feedback into decision-making processes. With this aim, GOA – the Governance and Policy Capacities Assessment Tool – has been developed. The GOA Tool is supposed to guide and assist Public Administration (in a process of self-evaluation) or external evaluators during processes of systemic assessments of intangible assets. The tool is designed in a general and flexible manner and can be adapted to the specific characteristics of each case. By this, it can be applied in different spatial contexts (community, local, regional, national) or policy fields (social inclusion, sustainable development, climate policies, innovation policies, etc.). The tool is based on existing tools e.g. the UNDP capacity measurement framework or approaches to organizational assessment. However, it has been extended and adapted to the European framework of regional policies. The GOA Tool consists of various components (see the figures on the following pages):

1) Four Dimensions of Regional Intangible Assets (each with a series of factors to be assessed). 2) Five Stages of Development regarding the evolution of the factors and dimensions. 3) An assessment framework linked to specific outcome and impact indicators in each case (optional). 4) A detailed assessment tool with diagnostic questions for each Factor in each Dimension and a graphic tool to visualize the assessment.

The GOA tool is still in the development phase and practical testing is currently being prepared (as of October 2014). The tool is thought to be an instrument for external evaluation, but also – after an introduction process – as a tool for self-assessment and learning, that can be used regularly. For its application in a specific case, a tailor-made methodology for data gathering (Interviews, survey, focus groups, observation, desk research and/or documentary analysis) and analysis (use of diagnostic questions, detailed qualitative description of the five stages of development for each dimension and factor, qualification of the system to be analysed according to the stages: excellent – very good – good – basic – poor) has to be developed. Although it is now envisioned for the European context of regional development, it might be easily applied in other geographic areas and even transferred to other thematic fields (climate change negotiations, education, health, etc.) in further stages.

13

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

Figure 6. Four Dimensions of Regional Intangible Assets

Source: Own elaboration

14

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

Figure 7. Five Stages of Development for Intangible Assets

Source: Own elaboration

15

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

Figure 8. Assessment Framework for Intangible Assets in Regional Development

Source: Own elaboration

16

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

Figure 9. Diagnostic Questions for the Assessment of the Dimension D1 Political Capacities

Source: Own elaboration

Figure 10. Example of an Assessment of the Dimension D1 Political Capacities

Source: Own elaboration

17

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

5. Conclusions This paper presents a new tool (GOA Tool) to help to assess, and therefore to improve and develop, intangible assets (governance suitability and policy capacities) in regional development processes. The tool has been designed especially for the European context of cohesion policy and similar complex regional processes. The analysis in the paper has shown that there are several approaches to assess intangible assets such as capacities, however, few existing methods can be applied directly to an overall systemic level (of a region or sector with a multitude of actors involved) or in an European setting. This paper offers a theoretical proposal of a new model that enables evaluators, public administration or other regional key players to assess the capacities that are needed to cope with complex societal challenges in multi-level and multi-agent playgrounds. The next step should be the practical testing of the GOA Tool and the development of additional indicators to assess the outcome and the impact of the existing capacities. An interesting case study would be the implementation of the GOA Tool in a specific region in order to assess the improvement of capacities in Regional Innovation Systems within the Smart Specialisation Strategies (RIS3) 2014-2020.

References 1 ADA (2011): Manual Capacity Development. Guidelines for Implementing Strategic Approaches and Methods in Austrian Development Cooperation. Austrian Development Agency. 2 BID-IDRC (2002): Evaluación Organizacional. Marco para mejorar el Desempeño. Banco Interamericano de Desarrollo. Centro International de Investigaciones para el Desarrollo. 3 Borgatti, S., Everett, M., Johnson, J. (2013): Analyzing Social Networks. SAGE London. 4 Burt, R. (1997): The Contingent Value of Social Capital. Administrative Science Quarterly. 42, pp. 339-65. 5 Cedefop (2007): Learning together for local innovation: promoting learning regions. Report edited by Bjørn Gustavsen, Barry Nyhan, Richard Ennals. Cedefop - European Centre for the Development of Vocational Training Luxembourg. 6 CIDA (2000): Capacity Development: Why, What and How. Capacity Development Occasional Series 1(1). Canadian International Development Agency. 7 Dibbon, D.C. (1999): Assessing the Organizational Learning Capacity of Schools. PhD Thesis. University of Toronto. 8 Drucker, P. (1993): Post-Capitalist Society, Harper Business, New York. 9 EC (2005): Institutional Assessment and Capacity Development Why, what and how? Aid Delivery Methods Concept Paper. EuropeAid. European Commission. 10 Fiorini A., Buscemi, V., Angori F. (2006) Study on Indicators for Monitoring Transnational and Interregional Cooperation Programmes. INTERACT Programme Secretariat, Vienna. 11 Giuffre, K. (2013): Communities and Networks. Using Social Network Analysis to Rethink Urban and Community Studies. Polity Press. Cambridge. 12 Grabow, B., Henckel, D., Hollbach-Gröming, B. (1995): Weiche Standortfaktoren. [Soft Production Factors] Stuttgart, 1995. 13 Grootaert, C., Van Bastelar, T. (eds.), Kahkonen, S., Krishna, A., Pantoja, E., Reid, C., Salmen, L. F., Shrader, E. (2002): Understanding and measuring social capital - a multidisciplinary tool for practitioners. Directions in Development. Washington, D.C.: The World Bank. Accessed 6 August 2014 at: http://go.worldbank.org/Z1K8SL5FK0 14 IAREG Project (2010): Intangible Assets and Regional Economic Growth. IAREG Project website www.iareg.org. Accessed 6 August 2014. 18

th

11 Biennial EES Conference “Evaluation for an Equitable Society”, 1-3 October 2014 Dublin

15 Kania, J., Kramer, M. (2013): Embracing Emergence: How Collective Impact addresses Complexity. In: Stanford Social Innovation Review. January 2013. 16 Kania, J., Kramer, M. (2011): Collective Impact. In: Stanford Social Innovation Review. Winter. 2011. 17 Kuhlmann, S., Shapira, P., Smits, R. (2010): Introduction. A Systemic Perspective: The Innovation Policy Dance. In: Smits, R.; Kuhlmann, S.; Shapira, P. (eds.): The Theory and Practice of Innovation Policy. An International Research Handbook, Cheltenham, UK (Edward Elgar), 1-22 18 Lember, V., Cepilovs, A., Kattel, R. (2013): Demand-side innovation policy in Estonia: rationales, limits and future paths. Working Papers in Technology Governance and Economic Dynamics no. 56. http://technologygovernance.eu/files/main/2013122012033838.pdf 19 Lundvall, B.-A. (ed.) (1992): National Systems of Innovation: towards a Theory of Innovation and Interactive Learning, London: Pinter Publishers. 20 Nemeth, L. S. (1997): Measuring Organizational Learning. Master Thesis. University of Western Ontario. 21 OECD (2009): Applications of Complexity Science for Public Policy: new tools for finding unanticipated consequences and unrealized opportunities. OECD Global Science Forum, Paris 22 Putnam, R. (1993): Making Democracy Work: Civic Traditions in Modern Italy (Princeton, NJ: Princeton University Press) with Robert Leonardi and Raffaella Nanetti. 23 Preskill, H., Parkhurst, M., Splansky Juster, J. (2014): Guide to evaluating Collective Impact. 01 (Learning and Evaluation in the Collective Impact Context) and 02 (Assessing Progress and Impact). Collective Impact Forum. FSG. Accessed 6 August 2014 at: www.fsg.org 24 Reflect&Learn Webpage (no year): Learning together about Organizational Assessment. Web-based Resources: http://reflectlearn.org/ by Universalia, IDRC/CRDI, The Rockefeller Foundation, accessed 6 August 2014. 25 Roth, A.M., Popescu A.-M. (2012): The development of intangible assets through the cohesion policy. MPRA Paper No. 40368, 6. August 2012. Accessed 6 August 2014 at http://mpra.ub.uni-muenchen.de/40368/ 26 Sabatini, F. (2007): The Role of Social Capital in Economic Development. University of Bologna. Working Paper n. 43. Accessed 6 August 2014 at: http://aiccon.it/file/convdoc/n.43.pdf 27 Suriñach, J., Moreno, R. (2011): The role of intangible assets in the regional economic growth. Investigaciones Regionales, No. 20/2011, pp. 165-193, AECR, Spain. Accessed 6 August 2014 at: http://www.redalyc.org/articulo.oa?id=28920705009 28

The World Bank (no year): Webpage www.worldbank.org/socialcapital

Social

Capital.

Accessed

at

6

August

2014

at:

29 UNDP (2008): UNDP Practice Note: Capacity Development. Website: www.capacity.undp.org 30

UNDP (2010): Measuring Capacity. _Measuring%20Capacity_English.pdf

http://sustainabledevelopment.un.org/content/documents/953015

31 Woolcock, M., Narayan, D. (2000): Social Capital: Implications for Development. Theory, Research, and Policy. World Bank Research Observer 15(2), pp. 225-249.

NOTES

1

These are characteristics which can be observed in systems and which make it difficult to analyze and understand them with linear approaches. For more detail on social systems and their characteristics, please see, e.g. Luhmann, N. (1984): Soziale Systeme. [Social Systems]. Suhrkamp, Frankfurt a.M.; − Meadows, D.H. (2008): Thinking in Systems: A Primer. Edited by Diana Wright, Sustainability Institute; − OECD Global Science Forum (2009): Applications of Complexity Science for Public Policy: new tools for − finding unanticipated consequences and unrealized opportunities. OECD, Paris. 19