03 Forss (bc/d)
30/1/02
2:40 pm
Page 29
Evaluation Copyright © 2002 SAGE Publications (London, Thousand Oaks and New Delhi) [1356–3890 (200201)8:1; 29–45; 021515] Vol 8(1): 29–45
Process Use of Evaluations Types of Use that Precede Lessons Learned and Feedback KIM FORSS Andante – Tools for Thinking, Sweden
CLAUS C . REBIEN COWI Ltd, Denmark
JERKER CARLSSON* Among practitioners, it is often said that there is much that is useful that takes place during the evaluation process. The literature on the utilization of evaluation, however, mostly refers to lessons learned and recommendations implemented after the evaluation. This article seeks to contribute to the debate by exploring further the process use of evaluations – the evaluation use that takes place before lessons learned are generated and feedback processes are initiated. Five different types of process use are identified: learning to learn; developing networks; creating shared understanding; strengthening the project; and boosting morale. Most resources in an evaluation process are spent in preparation and implementation. Very little is spent on feedback. A conventional way of improving evaluation use would be to shift resources from the preparation and implementation stages to the feedback stage. The article argues that in general, shifting resources from preparation and implementation to disseminating results would jeopardize the quality of the entire evaluation because it would ceteris paribus reduce the design, data collection and analytical quality. Instead it is argued that evaluation commissioners and evaluators should work explicitly to increase process use as the most costeffective way of strengthening the overall utility of an evaluation. K E Y WO R D S : empowerment evaluation; evaluation utility; learning; organizational change; participatory evaluation * Professor Jerker Carlsson was a good friend, dedicated colleague and inspiring coauthor of the present article. It is with great sadness that we – the two co-authors – honour his memory after he passed away at a much too early age in the autumn of 2001.
29
03 Forss (bc/d)
30/1/02
2:40 pm
Page 30
Evaluation 8(1)
Introduction The literature on the utility of evaluation mostly refers to the end result of evaluation: the conclusions and recommendations in the report. In this article the intention is to explore the process uses of evaluation, defined as the utility to stakeholders of being involved in the planning and implementation of an evaluation. Indeed, members of organizations testify that the actual process of preparing for and being involved in an evaluation can be very helpful (Patton, 1997, 1998). The end report may not be utilized, but the evaluation may still have been effective, e.g. it may change management thinking about future options; mobilize staff around a course of action; or reinforce the impact of a programme of social change. An evaluation may become part of the treatment, rather than just being an independent assessment of effects.
Purpose To explore the subject further it is necessary to develop a typology of different types of process use. These would be different from the usages often mentioned in literature (decision making, learning, control and conceptual use, etc.). Furthermore, we will explore how common different types of process use are, and which activities and institutional contexts they appear in. The purpose can thus be summarized: 1. to develop a typology that describes and analyses the process use of evaluation; and 2. to generate hypotheses regarding the context and activities relating to the process use of evaluations.
Method The development of the typology in this article is based on the three following methods of research. Firstly, it is based upon a comprehensive review of the literature on evaluation use and on studies and experience with evaluation processes within specific organizations. Secondly, it is based upon two specific case studies concerning the Nordic Development Fund (NDF) and Swedish Museums. Staff members were interviewed about the benefits and problems concerned with their involvement in the evaluation to find out whether this was a worthwhile activity, or whether the evaluation would be equally effective if out-sourced. A careful analysis was carried out to ascertain what the staff members actually learned from the evaluation, and what benefits resulted from their involvement. There was a deliberate attempt to verify whether the patterns of process use suggested by Patton (1997 and 1998) were present or not.1 Thirdly, in-depth analyses were carried out based on case study methodology of three evaluation processes with which the authors have been involved.
30
03 Forss (bc/d)
30/1/02
2:40 pm
Page 31
Forss et al.: Process Use of Evaluations
The Emerging Interest in Process Use Utility is a prime virtue of evaluations, indeed it is the first criterion of evaluation quality mentioned by the Joint Committee on Standards: ‘The utility standards are intended to ensure that an evaluation will serve the information needs of intended users’ (Sanders, 1994). Several manuals and handbooks urge evaluators to be practical and utilization-focused (Patton, 1978, 1982, 1997; Pawson and Tilley, 1997; Torres et al., 1996). Among the design features that will enhance the utility of an evaluation are the involvement of stakeholders in many aspects, such as identifying the problem, gathering data, and the design of a strategy to communicate the results. Nevertheless evidence from the field suggests that many evaluations are not utilized (Carlsson et al., 1999; Rebien, 1996; Thoenig, 2000). The institutional links between evaluations and decision making are weak (Cracknell, 1990) and it is difficult to prove that evaluations improve organizational learning (Samset et al., 1992). It is commonly believed that most evaluations end up gathering dust on shelves, and a number of scholars and critiques are raising the question: do we need all these evaluations? A more sophisticated analysis of evaluation utility may argue that evaluations can be used in many different ways. Even if the recommendations are not necessarily followed immediately, there might be other, more long-range effects. The knowledge generated in any programme context may be applied in a totally different context many years later. Hence, those who trace learning and utility with respect to evaluations confront a difficult task. People often forget where the knowledge and experience that they apply in decision making actually come from – and if a past evaluation played a role, that role is likely to have been forgotten. The discussion of evaluation utility belongs to a wider area of how information is disseminated and utilized in a social system (McMaster, 1996; Eden and Spender, 1998). There is a similar discussion with respect to the links between basic research, applied and development research, and the commercialization of innovation. Institutions involved in agricultural research are seeking ways and means to measure the impact of research findings on the practices at farms, through the process of reaching out to a dissipated system of individual – and independent – farmers, which is challenging (Forss, 1999). So, an evaluation can be used in many different ways. Vedung (1997) suggests that an evaluation can be used in seven different ways: instrumentally, conceptually, legitimizing, interactively, tactically, ritually, and as a process. The legitimizing and ritual uses of evaluation can – from a rational decision-making point of view – be called illegitimate; they have little to do with the search for knowledge, and recommendations for action based on new findings. Instrumental, tactical and interactive uses imply strategies to share results (or withhold some results) and engage in discussion on future courses of action with respect to the evaluation object. The conceptual use relates strongly to learning, i.e. changing a mental framework, which may or may not influence future thinking and decision making. With respect to each of these different ‘uses’ there is a field of reference: an established framework of books and articles. 31
03 Forss (bc/d)
30/1/02
2:40 pm
Page 32
Evaluation 8(1) Vedung argues that utilization must not necessarily take its starting point from the existence of a report. An evaluation report as such is not strictly necessary. Though many may see the word evaluation as synonymous with a mode of documentation – a report – this is a convention which is not supported by theory. The common definition of evaluation is ‘ a systematic inquiry into the worth or merit of an object’ (Sanders, 1994). The reader will note that this definition describes a process, an activity. Whether the activity will be documented and communicated via a written report is not a necessary requirement. Marra (2000) presents a good summary of how networking and the use of multiple channels of communication strengthen the use of evaluation findings. During their own work, the authors have found that people have often said that the findings were not particularly surprising; the effects or impact of the evaluated project were known about – or at least the results the evaluators found had been entertained as the possible effects. Recommendations presented by evaluators are very often within the scope of what people in the evaluated organizations would naturally consider as possible courses of future action. So given this predictability, what do they mean when they say ‘it was useful to go through the process’? Patton (1997) defines process use as the kind of usage that: is indicated by individual changes in thinking and behaviour, and program or organizational changes in procedures and culture, that occur among those involved in evaluation as a result of the learning that occurs during the evaluation process. (1997: 90)
Patton’s definition is meant to capture the distinction between learning that occurs as a result of the findings that are reported in the evaluation, and the learning that occurs during the process, irrespective of what is said in a report. But, as a definition, it is not without problems. It would actually be quite hard to verify process use, if the concept is not further explained, or pinned down with more specific examples. Patton (1997: 91) briefly defines four primary kinds of process use: 1. enhancing shared understandings, especially about results; 2. supporting and reinforcing the programme through intervention-oriented evaluation; 3. increasing participants’ engagement, sense of ownership, and self-determination; and 4. programme or organizational development. The categories throw light on many of the issue of process use, but it is not always clear what distinguishes them from the use of findings. In the categories mentioned, it may seem that process use is closely intertwined with sharing the findings of an evaluation. In a follow-up to the above distinctions, Patton (1998) suggested additional types of process use, referred to as ‘learning to learn’, ‘qualitative insights’, and possibly ‘goal displacement’. Patton (1998) does not actually speak of categories of process use in the same rigorous sense as in the previously referred book (Patton, 1997). Patton’s work touches on essential questions, and there is a need to further develop the typology, and explore varieties of process use. Yet another issue for 32
03 Forss (bc/d)
30/1/02
2:40 pm
Page 33
Forss et al.: Process Use of Evaluations further exploration relates to the universality of process use. Can process use, however we define it and whichever subcategories we distinguish, be found on all and every type of evaluation? Is it even desirable to pursue process use in all evaluation settings? Assuming that some evaluations are initiated primarily for reasons of control and audit, would that diminish the potential for process use? Is process use mainly something that can be pursued with respect to participatory evaluation, or so called empowerment evaluations? Is it important to discuss the purpose of the evaluation before we start discussing how process use can be strengthened? In the analysis of the empirical examples below these issues are discussed.
The Various Ways to Use an Evaluation Process In the study concerning the NDF, we identified the following five types of process use. Learning to Learn Patton (1998) wrote that the evaluation field has its own particular culture, building on norms and values that evaluators take for granted, but which may be quite alien to people embedded in the culture of another profession. Patton (1998: 226) suggests that these values include ‘clarity, specificity and focusing, being systematic and making assumptions explicit, operationalising programme concepts, ideas and goals, separating statement of fact from interpretations and judgments’. It could of course be argued that in modern organizational settings, these values are quite universal; however, it is one thing to subscribe to a value in theory, quite another to act according to that particular logic. Even if values such as those mentioned by Patton are subscribed to by many, most would find that working life offers limited opportunities to act according to these values in practice. The values of evaluation are values conducive to enquiry. These values support a structured way of thinking about reality and generating knowledge. There are many ways of generating knowledge; such as with the help of guesswork, supernatural insight, divine inspiration, and transcendence. Historically, there can be no doubt that the application of a scientific method is the cheapest and richest way of generating knowledge, and the systematic conduct of an evaluation is a superior way of learning. To engage in evaluation is thus also a way of learning how to learn. Within the vast amount of literature on evaluation theory there are several articles on how to learn from evaluation. Patton (1998) can be read to suggest that this is a misguided perspective – evaluation is actually synonymous with learning; it is the process which generates insight and knowledge. Failure to learn from evaluation actually only refers to failure to disseminate the findings, to organize the consumption. Those who learn from evaluation are those who do the evaluation, very few others learn to any significant extent. But those who do learn, may learn quite a lot, and they do so in the process of inquiry. The staff members of NDF unanimously testified to the value of learning to learn, to organize a systematic inquiry, to patiently search for opinions, facts, 33
03 Forss (bc/d)
30/1/02
2:40 pm
Page 34
Evaluation 8(1) evidence of impact etc. They learnt to assess responses, to search for explanations regarding why people profess certain opinions about a programme, and how that relates to self-interest. They learnt to seek information from supplementary sources, to crosscheck for validity and reliability. To engage in evaluation can thus be a way of strengthening organizational learning – something which is very much in fashion. Developing Professional Networks NDF staff members often mentioned that being part of the evaluation allowed them to meet others whom they otherwise had little interaction with. They referred to clients, competing organizations, stakeholder groups, as well as members of their own organization. The importance of this is partly due to the peculiar nature of NDF financing. The NDF provides finances for part of a loan agreement, where other financiers play a dominant role. When a financial package is put together, it is important to have personal contacts. These lower transaction costs, by establishing trust and facilitating communication. In practice, members of staff of the different banks have a heavy workload and they seldom meet unless they have to, and then under severe time constraints. The evaluations conducted by NDF personnel often meant that they had to visit the other banks, as well as ministries of finance in borrowing countries, peers in other organizations, researchers, etc. Many of the personnel met during the conduct of the evaluation turned out to be useful contacts for other occasions. The next time a loan is negotiated, there is a face to connect to the mail address. Also, in the course of ordinary duties, people need information, and the new contacts added in the process of evaluation are possible sources of knowledge. In the present information society, building and maintaining a relationship are of vital importance for professional success and similarly an organization would need members with extensive networks (Peters, 1994). Evaluation appears to be a way of rapidly interacting with many people, and hence an opportunity to build networks. Creating Shared Understandings In the process of analysing the impact of a number of museums, the authors distinguished between the following four different roles of a museum. • Acquiring and keeping things of the past; • Maintaining these in proper conditions for future generations; • Conducting research on the use and meaning of items in the museums’ collections; and • Serving the general public with exhibitions, tours, displays, lectures and the like. Different personnel categories were found to have quite different priorities. Whereas curators saw the museum’s main function to be in the building and maintenance of collections, the political leadership on museum boards often wanted to give priority to using the collections for public purposes. When considering how to spend an additional sum of money or choosing where to make budget cuts, 34
03 Forss (bc/d)
30/1/02
2:40 pm
Page 35
Forss et al.: Process Use of Evaluations they had quite different priorities. The evaluation made these differences explicit. Naturally, this did not take away dissent, but it may have led to a more successful arbitration of interests. It helped people understand each other’s motives, and to some extent also to respect the differences. This communication and the mutual understanding which developed were quite unrelated to the thrust of the evaluation report. However, there are many different aspects of how communication influences evaluation work, internally on evaluation teams as well as between the evaluator and various stakeholders. It would seem that the usefulness of evaluation hinges directly upon the quality of communication in evaluation exercises, and that such communication requires skills and evaluator experience. It is therefore worthwhile discussing the precise content of such skills and how they can be developed. A comprehensive discussion is beyond the scope of this article but it is suggested that the skills include intangible qualities such as cultural sensitivity and applying what in German is termed Fingerspitzengefühl. This term describes being sensitive to any given situation in which the evaluator may find him or herself, and to how others may react to what one says or implies. Regardless of how intangible these qualities may be, the consequence of not having them will rapidly become manifest. The best way of acquiring such skills or personal qualities is through experience. Evaluation is defined by its practice. And evaluators are best trained through apprenticeship, i.e. through being involved as team members in evaluations with team leaders who can act as role models and mentors, and who can gradually expose the team member to ever increasing challenges. In addition to the apprenticeship model, the management training industry and business schools run many courses, seminars and entire degrees in which one can acquaint oneself with the theory of intercultural skills and communication. The purpose of this should be clear: to aid good communication and avoid misunderstandings, and to promote interaction and exchange of knowledge and experience. In addition to the individual being better equipped in terms of communicative skills, it is important, however, that structural barriers to communication are kept at a minimum. The usefulness of the evaluation tool in the future also hinges on its ability to provide timely and easily accessible answers to questions that must be answered in order for a project, a programme or an organization to progress. The tendency towards increasingly complex evaluation designs, with multiple teams and lasting for long periods of time, does not exactly facilitate communication, nor does it promote the usefulness of the evaluation tool in terms of being able to provide input to change processes. Strengthening the Project Evaluations are often undertaken to assess past performance, to analyse whether goals have been reached or not. In reality, the projects that are financed as specific organizational frameworks for action are embedded in organizations where many other activities occur at the same time. When the evaluator appears on the scene, he or she may often find that people are reminded of the project being evaluated. 35
03 Forss (bc/d)
30/1/02
2:40 pm
Page 36
Evaluation 8(1) During the process of an interview, what the project was (and is) about will be re-examined by the interviewees. The result is that they come out of the meeting with the evaluator with a clearer understanding of the project, and possibly with a new resolve to achieve the project’s aims. Irrespective of what the evaluators write in their report, the purpose the project was meant to serve may thus have been strengthened. This effect is encountered particularly often in evaluations of technical assistance and of education. In a typical example, managers who have been through a short training programme come back to work and are immediately overwhelmed by routines. The ideas – and the resolve to use them – learnt during the training are soon forgotten. However, a meeting with an evaluator, who follows up the impact of training, can serve to remind them of their intentions to put the training to use, and if undertaken by a skilful evaluator, these intentions may actually be strengthened. Management training programmes often include an exercise whereby the participants have to think of themselves as change agents, and set targets for things they want to accomplish when they are back at work. They are often visited by some of the faculty six months later, or they reconvene with the whole course, to assess what they have achieved. Through such processes two birds are killed with one stone: evaluators get prime data on the changes brought about by the programme, and the risks that the programme is forgotten once the participants are back at work are reduced. Defining, gathering and analysing evaluation data become synonymous with strengthening the impact of the training programme. Patton (1998) calls this ‘evaluation as an intervention’; the evaluation becomes an intentional intervention supporting programme outcomes. It is apparently more common in many social and health-related programmes. Here a follow-up meeting with a patient, a beneficiary or a person who has been through some kind of social programme, is taken as both an opportunity to collect information, a reminder of the results that are expected, and possibly a point at which to make new decisions. Collection of data is built into the programme process in ways that also strengthen the achievement of objectives. There is yet another kind of process use that strengthens the project, but which occurs through a different mechanism. Much of professional work today occurs under considerable time constraint. The productivity increases that have been witnessed over the past few decades have come at the cost of redundant time. As redundant by definition is ‘redundant’, the cost may appear low. However, general systems theory advises that redundancy is a way of adding variety in a system, and of increasing flexible adaptation (Morgan, 1986). One of the things that happens during an evaluation is that the work flow – or whatever process is being evaluated – is interrupted. Imagine an evaluation concerned with the impact of a programme to develop municipal democracy where the evaluators are a team of political scientists. During the course of the evaluation, the evaluation team will interact closely with politicians and bureaucrats. The way the team formulates questions, discusses activities and listens to experiences, may influence activities at the project level. 36
03 Forss (bc/d)
30/1/02
2:40 pm
Page 37
Forss et al.: Process Use of Evaluations Evaluators do not necessarily share their wisdom and experience; the process is rather more complex. It involves concentrated interaction with someone who is interested, who thinks sharply about the issues, and who may be ready to discuss difficult problems of resource allocation, strategic choice and responses. This is worthwhile in its own right – and can lead to changes quite separate from the topics that arise out of the evaluation’s terms of reference. The nature and extent of such changes may simply not be visible, or even relevant, for the evaluation report. Boosting Moral Brunsson (1989) stated that evaluation is a ‘depressive’ activity. Evaluation leads us to question, to be sceptical, to reassess – thus perhaps also to be distanced. Evaluators are not well known for good humour and enthusiasm, which are both essential ingredients in successful action. As Brunsson says, action thrives on commitment and enthusiasm. Evaluation, on the other hand, is an intellectual activity concerned with problems. In that sense evaluation is ‘depressive’. Practitioners who are confronted with an evaluation team may recognize these sentiments. It is therefore surprising that the staff members of NDF often mentioned that their own engagement in evaluations increased their commitment to the organization and to its activities. Taking part in evaluations generated enthusiasm and commitment, rather than the opposite. It served to boost morale. The paradox can be explained by looking at the role of the organization in relation to the activities that are evaluated. The staff of NDF were not evaluating their own work processes. They evaluated their clients, the borrowing organizations in developing countries. NDF is, in essence, an intellectual and reflective organization (in Brunsson’s terminology). Furthermore, the day-to-day activities of the staff members are quite abstract and far removed from the down-to-earth implementation of projects. The evaluation process allowed them to see reality at close hand. It reminds them of the purposes they work for, and allows them to explore the relationship between their own organization and the development impact that is expected. Even though that confrontation with reality will sometimes lead to insights about the shortcomings of projects, mistakes and abuses of results, the net effect appears to boost morale. The question is whether that happens as a side-effect of other characteristics (travelling, network building, changing pace, learning to learn) or whether it is an independent aspect of process use. In practice, the effect of boosting morale is likely to be both an independent category of process use, and something which is influenced by other aspects of process use. The first reason is that organization theory, particularly the thoughts reflected in Brunsson’s work and others following him, emphasize the negative, the depressive and paralysing consequences of evaluation (and related) activities. It is actually one of the major categories in thinking about the nature of organizations. Hence, to confront and question that view of evaluation, we put forward the opposite argument – that evaluation is a way of boosting morale, strengthening action and generating enthusiasm. Resulting issues that need to be addressed 37
03 Forss (bc/d)
30/1/02
2:40 pm
Page 38
Evaluation 8(1) relate to whether this happens often, whether it can be designed to occur more frequently, and whether it is a desirable aspect of process use. The second reason to define this as an independent category is that even if boosting morale is a possible side effect of building networks or strengthening a programme, etc., it should also be noted that it is not the same as these processes. Morale, a supportive organizational culture, belongs to the intangible assets of an organization. It is part of the values that all agree are fundamental, but which elude description and measurement. Nevertheless, those who have entered a confused and dispirited organization will certainly recognize how tangibly different that is from a focused, inspired and committed organization. To avoid this potential aspect of process use because it is vague, difficult to understand and measure, would result in the fallacy of ‘doing only those things that can be measured’. In this context it may also be relevant to recall the ‘Hawthorne effect’. The process of evaluation may produce a similar effect. Evaluations are carried out by well-managed and professional organizations. Being exposed to an evaluation is a sign that one is part of such an organization. The fact that attention is shown, the project is investigated, viewpoints are listened to and data are collected could presumably give rise to similar positive effects as was seen in the Hawthorne experiment.
Forming a Typology from the Five Types We have identified five different types of process use: learning to learn, developing networks, creating shared understanding, strengthening the project and boosting morale.2 The first issue to consider is whether the typology is exhaustive or whether we have missed important aspects of process use. Patton mentions two other kinds of process use: ‘qualitative insights’ and ‘what gets measured gets done’. Gaining qualitative insights is similar to some of our categories, namely creating shared understanding and strengthening the project. It seems to the authors that these two categories are quite different, though both build on some kind of qualitative insight. As a general phenomenon, a broad category called ‘qualitative insight’ would not be distinct from learning to learn, but the latter is distinct from creating shared understanding and strengthening the project. Patton’s category ‘qualitative insights’ can thus be seen as a combination of two, if not three, of our categories. The category ‘what gets measured gets done’ is a well-known side effect of evaluation and performance management. If it is known beforehand that an evaluation team is coming to check fire safety routines, foremen may carry out some extra fire drills, to make sure that standards are high. Therefore the evaluation will show that standards are high, and they are, but this may be due to the knowledge of the forthcoming evaluation. The fire drills replace other activities, and it is not known whether those activities would actually have been more important. It could be argued that this effect is the negative aspect of using the evaluation to strengthen the project. If the aim was to strengthen preparations for fire, then spreading the knowledge that this would be evaluated could be a way of accomplishing the effect. The fact is that an evaluation process is not neutral, it will 38
03 Forss (bc/d)
30/1/02
2:40 pm
Page 39
Forss et al.: Process Use of Evaluations reinforce and strengthen some aspects of the organization, presumably at an opportunity cost of time and money. Whether the effect is for better or for worse should be an aspect of the category, rather than a category in its own right. The next question is whether some types in the typology are redundant. Do they overlap each to such an extent that the list should be reduced to three or four kinds of process use? The types were defined as quite distinct from each other. They involve different kinds of learning processes for different purposes. Nevertheless, the categories are not independent of each other. In particular, the category of boosting morale could be seen as a secondary effect of learning to learn or developing networks. We have opted for treating boosting morale as a separate category, as it can be observed without being an effect of the others. It can occur together with them, or without them, and their presence does not necessarily boost morale either. However, it seems as if the different categories of process use will affect – or are likely to affect – stakeholders in an evaluation process in different ways. Table 1 below summarizes which actors are the most likely to benefit from each kind of process use. The table suggests that, primarily, those who do the evaluation will benefit; this refers to those who set the targets of the research, select the appropriate methods and collect and analyse data. The evaluators do not necessarily need to be an external team. As the evidence from the NDF case study indicates, this is a strategic choice. If the extent of process use is to be maximized in an organization, staff members should take part in evaluation work as much as possible. A high degree of process use is something that can be planned, designed and promoted. The choice of method will also have an impact on process use. It would seem that several aspects of process use depend upon personal communication; various kinds of interviewing methods would therefore seem more likely to generate it. Documentation analysis and large-scale surveys would probably be the methods that generate the least opportunities for process use. To further complicate the subject, it is also possible that the combination of personnel – external evaluators, stakeholders and others whom the evaluators interact with – will affect process use. An evaluation based on interviews with management in an organization may generate higher opportunities for micro-level change if the evaluators are a team of experts recruited in a peer group. If the evaluation effort is directed at technical levels of an organization, professionals of a more specific nature will probably increase the chances of process use. Table 1. Actors who will Benefit from Process Use Category of process use
Actors who will benefit
Learning to learn
To some extent those who commission the evaluation, and who furnish information, but primarily the evaluators Primarily the evaluators Primarily those who furnish information Primarily project management Primarily the evaluators
Developing networks Extending communication Strengthening the project Boosting morale
39
03 Forss (bc/d)
30/1/02
2:40 pm
Page 40
Evaluation 8(1) In sum, process use can be described as an outcome which may vary quite significantly. It is an outcome that can be seen as positive or negative, and comes at an opportunity cost in the use of time and money. It will affect the stakeholders in an evaluation process differently. High process use by some may imply a low use by others, or even a use that may hold a negative value.
Can Process Use be Facilitated and Strengthened? It is well known that the utility of an evaluation process can be increased by extending communication when the results are presented. Working on the report, finding ways of supplementing a report with other presentations and connecting to decision makers are all worthwhile efforts. However, one practical implication of such suggestions is that budget resources are shifted from earlier stages in the evaluation process to latter stages. In a typical evaluation process, most of the money is spent on data collection and analysis, a minor share is spent on preparation for the evaluation, and even less is spent on disseminating the results. A rough approximation may be that 20 percent of the resources are spent in anticipating and formulating the evaluation; 75 percent of resources in conducting the evaluation, and 5 percent in disseminating the results. A high quality evaluation process is defined in terms of utility, feasibility, propriety and accuracy (Sanders, 1994). Whereas there can hardly be any doubt that a shift of resources from earlier to the last phase in the evaluation would enhance utility, a likely outcome would be that another quality aspect would suffer. Furthermore, there is a hypothesis that more learning takes place before and during the evaluation, than after. This does not mean that upon completion the evaluation is of no use; it does not necessarily collect dust on a shelf! But the report itself will normally provide little new insight. Most discoveries and new knowledge have been consumed and used during the evaluation process. The report merely marks the end of the evaluation process. It is perhaps not surprising that most of the learning occurs during the evaluation process at the time when most of the money is spent, rather than afterwards. But is it possible to increase the overall evaluation use without shifting resources from one evaluation stage to another? Could a marginal increase in utility be achieved at a lower cost, if the money is spent in preparation and implementation, rather than in disseminating the report? Indeed, can process use be traded off against use of findings and recommendations? Are they worth an equal amount? Can they even be compared? Two cases of process use are discussed below in relation to how process use was, or could have been, strengthened.
Case 1 – Enterprise Development A concrete example of process use is an evaluation of a small enterprise development programme in a county in Denmark. The programme was designed to assist small companies with less than 50 employees to access training and skillsupgrading courses for their staff. The programme hired six consultants with the 40
03 Forss (bc/d)
30/1/02
2:40 pm
Page 41
Forss et al.: Process Use of Evaluations task of identifying companies, working with them to identify training needs and identify the appropriate courses and vocational training centres where staff might acquire the necessary training. We – the evaluators – followed the programme for two years. The programme funders – the EU Commission, the Danish county and Government and a number of vocational training centres – wanted the evaluation to document accountability: whether taxpayers’ money had been well spent with good results. The final evaluation report after the programme was over shows the consequences of the programme to the companies: how many staff members were trained; what types of skills did they acquire; what did it mean to the company in terms of product development, changes in working routines, turnover, etc. But the real value of the evaluation was in its process use. The six consultants hired to the programme were convinced from the beginning that some of the barriers that exist in small companies for sending their staff for retraining and skills-development courses could be overcome. Small companies do not have staff training plans or personnel departments, and are even less likely to have human resource development functions or training budgets. Often staff members’ functions are so specialized that if even a single staff member is not present because he/she is sent for training, the production process will more or less be stopped. But these barriers did not discourage the consultants; other barriers did. The vocational training centres were co-sponsors of the programme. In addition to overall programme goals, their own interest was to sell courses to the companies. However, the programme consultants soon discovered that courses available at the training centres did not match all the priorities and training needs of the companies. Early in the life of the programme, the evaluation was able to document that this was the case. And although the vocational training centres were very dissatisfied, the consultants and the programme steering group – using the evaluation in the process to provide legitimacy – were able to identify additional providers of training courses which could fill the training needs gaps. In a similar way, the evaluation process was instrumental in changing the scope of the programme. It had been designed as a vocational training programme, but one of the consultants especially found this to be a very narrow approach to developing human resources in small companies. Developing skills of an individual staff member would, in his mind, have little effect on the company, unless linked with devices such as technological developments and changed working procedures, company work plans and strategies. Through dialogue with the evaluation team, and his five fellow consultants, a changed approach was developed during the first half of the programme period. The scope was changed in order that the identification of training needs, development of training plans and identification of training providers were supplemented with assistance and advice in areas completely new to most small companies. Training and training plans were linked to organizational and management development, identification of necessary changes in working procedures and the development of overall, strategic company plans. The process use of the evaluation led to programme 41
03 Forss (bc/d)
30/1/02
2:40 pm
Page 42
Evaluation 8(1) changes. And it was the new features of the revised programme that companies gave as the reason for assessing the programme to have been a success.
Case 2 – The Role of Experts in Technical Co-operation A number of overseas development agencies together commissioned an evaluation to analyse the effectiveness of expatriate experts on aid programmes. The evaluation was concerned with how expatriates are trained, receive job descriptions and the like. There was a general concern about details such as: an appropriate duration of an assignment abroad; type and extent of cross-cultural management skills; the difficulties of returning to a career at home; and the problems of accompanying spouses. The evaluation was primarily commissioned to learn about managing large technical assistance programmes. The recommendations were expected to feed into the decision-making mechanisms of the agencies that had commissioned the study. It was directed at senior management, in particular directors of personnel and recruitment. When the evaluation was completed, it was widely published, and the information requested did reach the target groups. The method chosen by the evaluators was interviewing expatriate advisers and their counterpart staff on a number of projects. In one such interview, the evaluators spoke to counterpart staff in an organization. They discussed personnel policies, and asked how the foreign experts were viewed. It was no surprise to find out that they were quite resented; arrogant behaviour, closed systems of decision making, reluctance to share knowledge and train local staff, high turnover, and extravagant living conditions were the main reasons. During the interview it turned out that several more expatriate experts were on their way; among them a financial adviser. The evaluators could see that the financial system was in shape, and that there was no need for assistance in that field. In fact, the local manager, who had been in charge for 15 years, was well educated and respected by others, did well, but he planned to leave as soon as the experts arrived. So, the assistance would destroy existing competence rather than help. The evaluators highlighted this problem to the aid agencies. Luckily, in this case there was a swift response, and recruitment for a financial expert was halted. The evaluation as a whole was of course not concerned with this particular project, and the main findings, conclusions and recommendations did not reflect this particular decision. But the process of evaluation generated change, it fed advice directly to decision makers, which led them to change a course of action.
Conclusion In addition to the generally accepted notion that evaluations, through feedback processes, add new knowledge to organizational/programme development after the evaluation, there is in fact much that is useful that takes place during the evaluation process. In the article we have defined this as process use. A general example of process use is the well-known reaction of stakeholders when the final report is being presented: ‘Your suggestions are fine. However, we 42
03 Forss (bc/d)
30/1/02
2:40 pm
Page 43
Forss et al.: Process Use of Evaluations have already adopted procedures along the lines that you suggest, so your recommendations do not in fact take account of the latest developments’. Hence, the evaluation has sparked changes to the programme/organization during the evaluation process, making the report itself obsolete. In the article, we identify five different types of process use. These extend the categories of process use introduced by Michael Patton (1997 and 1998) and distinguish the actual experiences more sharply from the use of findings and recommendations. Taken together, they occur irrespective of the findings and recommendations that emerge through the collection, analysis and presentation of data. They reside in the process of evaluation, and they occur as a side effect of this unfolding process. Whereas some kinds of process use benefit those who carry out the evaluation, other kinds benefit those who are target groups for an intervention, and yet other kinds of process use may benefit those who commission the evaluation, or indeed the project management. At first sight, it seems as if most of the positive effects of process use will be experienced by the evaluators. But it is important to note that who the evaluators are does not affect this; in fact, internal evaluators may have a high process use, as may external evaluators. This could be an argument in favour of participatory approaches to evaluation – to spread the utility of evaluation more evenly and widely. We do not doubt that process use exists, but we do not know how common it is. Additionally, it is difficult to ascertain whether it is a more common use of evaluation than the traditional use of lessons learned. But, since so many suggest that the use of lessons learned is severely limited, it may not be an exaggeration to suggest that in total, the different kinds of process use suggested here make up an equally important aspect of evaluation utility as the use of findings and recommendations. Hence there is a need to know more about process use, and to verify whether it occurs as often as some like to believe it does. The review of empirical examples suggests that process use occurs due to luck: it is a question of chance. While not underestimating good fortune, we suggest that process use could be planned and designed. Utility appears to be a weak spot in evaluation quality. The quest for utility is often focused on increasing the use of lessons learned, which implies shifting resources from implementation to the presentation and dissemination of results. If no resources are added, this will lower the quality of the other evaluation stages. Instead, experience with process use may indicate that further strengthening of data collection and analyses is needed in order to plan deliberately for use during the evaluation process.
Notes 1. It is inevitable that there is a positive bias in this approach. We have looked for instances of use; we have not been engaged in a critical examination of whether evaluations are used or not, or even whether process use occurs as often as it could. The analysis suggests possible types of use – it is not a critical examination of whether these actually occur. 2. This is a preliminary list of different process uses and is heavily indebted to Patton’s first identification of kinds of process use (1998). The types are only possibilities; they
43
03 Forss (bc/d)
30/1/02
2:40 pm
Page 44
Evaluation 8(1) do not necessarily or automatically occur. Maybe they do not occur at all; maybe some are more common than others. Further empirical research is needed in this area.
References Brunsson, N. (1989) The Organization of Hypocrisy. New York: John Wiley & Sons. Carlsson, J., M. Eriksson-Baaz, A. M. Fallenius and E. Lövgren (1999) Are Evaluations Useful? Cases from Swedish Development Cooperation. Stockholm: Sida. Cracknell, B. (1990) EEC Commission and Member States Review of Effectiveness of Feedback Mechanisms. Brussels: European Commission. Eden, C. and J. C. Spender (1998) Managerial and Organizational Cognition. Theory, Methods and Research. London: Sage. Forss, K. (1999) Questions and Methods in Impact Assessment; Outline of an Analytical Framework to Study Impact from Information and Communication. Wageningen: CTA. McMaster, M. (1996) The Intelligence Advantage: Organizing for Complexity. Boston, MA: Heinemann. Marra, M. (2000) ‘How Much Does Evaluation Matter? Some Examples of the Utilization of the Evaluation of the World Bank’s Anti-Corruption Activities’, Evaluation 6(1): 22–36. Morgan, P. (1986) Images of Organization. London: Sage. Patton, M. Q. (1978) Utilization-focused Evaluation. London: Sage. Patton, M. Q. (1982) Practical Evaluation. London: Sage. Patton, M. Q. (1997) Utilization-focused Evaluation. The New Century Text, 3rd edn. Thousand Oaks, CA: Sage. Patton, M. Q. (1998) ‘Discovering Process Use’, Evaluation 4(2): 225–33. Pawson, R. and N. Tilley (1997) Realistic Evaluation. London: Sage. Peters, T. (1994) Crazy Times Call for Crazy Organizations. London: Pan Books. Rebien, C. (1996) Evaluating Development Assistance in Theory and Practice. Aldershot: Avebury. Samset, K., K. Forss and O. Hauglin (1992) Learning from Experience – a Study of the Feedback from the Evaluation System in the Norwegian Aid Administration. Oslo: Ministry for Foreign Affairs. Sanders, J. R. (1994) The Program Evaluation Standards. Joint Committee on Standards for Educational Evaluation. London: Sage. Thoenig, J.-C. (2000) ‘Evaluation as Usable Knowledge for Public Management Reforms’, Evaluation 6(2): 217–29. Torres, R. L., H. S. Preskill and M. E. Piontek (1996) Evaluation Strategies for Communication and Reporting. London: Sage. Vedung, E. (1997) ‘Utvärdering och de sex användningarna’, in B. Rombach and K. och Sahlin Andersson (eds) Från sanningssökande till styrmedel. Moderna utvärderingar i offentlig sektor. Uppsala: Nerenius & Santérus Förlag.
KIM FORSS is the founder of Andante – Tools for Thinking, an independent centre for research and competence development in the fields of evaluation and organization research. He has published work in the following areas: technical cooperation, organizational learning, evaluation research and UN reform. Please address correspondence to: Andante AB, Bisp Thomas v. 8, 645 34 Strängnäs, Sweden. [email:
[email protected]]
44
03 Forss (bc/d)
30/1/02
2:40 pm
Page 45
Forss et al.: Process Use of Evaluations CLAUS C. REBIEN is Chief Projects Manager at COWI Ltd, the largest Danish consultancy. He has led numerous evaluations in Eastern Europe, Asia and Africa. In recent years his international experience has been applied to evaluations in the health and social sectors in his home country. Please address correspondence to: COWI Ltd, 2 Parallelvej, 2800 Kgs. Lyngby, Denmark. [email
[email protected]]
JERKER CARLSSON was a professor at the Institute of Peace and Development Research at the University of Gothenburg, and he was a co-founder of the Andante group of companies. He published work on development economics, economic history and the organization of partnerships and roles in development co-operation. Professor Carlsson passed away in September 2001.
45
