and Leviton (1991: 227-228) and Weiss and Rein (1991: 143). They discuss very thoroughly the idea that the function of organizations and projects has been ...
1 EVALUATION METHODS AND COMPARATIVE STUDY Prof. Pirkko Vartiainen University of Vaasa, Department of Public Management ABSTRACT The aim of this presentation is to discuss evaluation as a tool of a comparative research process. My starting point is that evaluation research is not purely comparative in nature but yet often contains elements which will forward the comparative analysis. In my presentation the analysis will focus on the following question: What kind of roles do evaluation methods have in the field of comparative analysis?
1. Comparative evaluation To be able to answer the question above I have to discuss several preliminary questions. My starting point is that comparative evaluation research has to fulfil requirements set to both comparative research and evaluation research. However, my aim is not to give long, separate quotations from the methodological literature of evaluation and comparison, but rather to try to combine these two approaches to the abstract cross-twine. To be able –at least to some degree- to illustrate this “twine” I will in my presentation analyse different types of evaluation on the basis of four principles (Table 1). These principles are briefly defined in chapters 1.1. – 1.4. They are as follows: - selection of the evaluation object - the level of comparison - the conceptual comprehension, and - the analysis of (evaluation) research results. In table 1 I have formed a frame on the basis of which the examination of the characteristics of comparative evaluation will proceed. The different types of evaluation included in the analysis will also appear on the table. As a starting point to the selection of these types I have used the classification from Chambers et. al (1992: 8-9). In this classification the evaluation research has been divided into five main categories according to the content and the essential viewpoints of the evaluation. When necessary, one or two methods are used in my presentation as examples to present each evaluation category. Table 1. The frame for the examination of comparative evaluation Principle Evaluation category Outcome and goal attainment evaluations
Selection of the evaluation object
The level of comparative evaluation
The conceptual comprehension
The analysis of evaluation research results
2 Cost oriented evaluations Implementaion studies Consumer satisfaction studies Needs studies
1.1.
Selection of the evaluation object
The selection of the evaluation object is an essential issue in comparative evaluation. There are several possibilities in selecting the evaluation object and the literature does not always give an unambiguous answer to it. We can nevertheless mention a few classical examples. According to Argyris (1962), in the context of evaluation processes one has to decide whether to evaluate individual action, behaviour, or contentment; Pennings – Goodman (1979) and Manns – March (1978) suggest concentrating on the evaluation of the co-ordination and goal-achievement of different organizational units, and Weiss (1972) speculates whether organizations should be evaluated as entities or merely on the programs they practice. The definitions of the evaluation object presented above bring up interesting questions, particularly when they are considered from the point of view of comparative evaluation. Evaluation connected with the internal structures and functions of organizations contains elements of individualism, behaviour, and contentment. For example in service organizations the diversity of interaction with customers sets requirements both on the professional competence of the employees and the ability of the working community to function properly. Therefore, one has to take into account the organization’s social environment and its internal structures and systems while doing a comparative evaluation. In practice this means that comparative evaluation process has to deliberate factors like; resources, implementation, results, social norms, and the role of the customers. By examining these factors the evaluation process analyses organization as an open system. This means that an evaluation of mere internal function of an organization cannot be considered as a sufficient basis for a study. During the evaluation process one has to take into consideration also the environment of the organizations and its effects.
1.2.
The level of comparison
When defining the level of comparative evaluation one has to decide the scope and the principles with which the evaluation is possible to accomplish. The units of comparison may be, for instance, political as well as organizational systems or the sub-systems of these. The level of the comparison may be either internal, (e.g. comparisons between local governments, intra-organisational comparisons) or
3 comparisons between the different systems (e.g. international comparisons, private vs. public sector comparisons). A third strategic choice connected with the level of comparative evaluation is concerned with the similarity and/or dissimilarity of the evaluation object. This is directly comparable with the already classical question: in which circumstances can different cases be compared to each other? Several researchers have already discussed comparability, so I will not get into detail with that, but will merely refer to authors such as Zeldith 1971, Peters 1988, 1998, Ragin 1989, and Rose 1991.
1.3.
The conceptual comprehension
When evaluation is used as a tool of comparison the key issue for success is probably the conceptual comprehension which is connected to both methods. It is crucial for a relevant comparison that the concepts will be clearly defined so that different parties (researchers, target groups, politicians etc.) can interpret and understand the concepts in question in the same way. The same logic is valid in evaluation. The definition of the context of the results and the operation of indicators and criteria play a significant part in the interpretation of the evaluation results. The definition of the concepts used in the study is important from the point of view of the comparative evaluation, since comparison heightens further the level of the analysis. The level will go up, particularly because in a study based on a comparative evaluation the concepts will be used both during the evaluation and the analysis, as well as during the comparison of the results of the analysis. As a concrete example of the contextual difference of meaning between concepts I could choose for example the article of Cheetham et.al. (1998: 16) in which they are well aware of the differences of the conceptual context of poverty in different countries: “Further problems of comparability concern different national manifestations of such apparently common concepts as poverty. There is a large difference in what this means in the US, UK, and Scandinavia…” Even this one example shows clearly that one has to spend time and effort to examine and standardize the conceptual contexts in the process of comparative evaluation. This is especially true when comparing organizations or countries where cultural factors (e.g. differences between the unofficial and official cultures of systems and/or organizations) mould the conceptual contexts.
1.4.
The analysis of evaluation research results
In comparative evaluation the analysis of research results is of course dependent on the method but also on the level of the evaluation. I assume that comparative evaluation produces comparable information most efficiently when the units which are analysed are as similar as possible. This assumption is based on the idea that it is easier to form reliable evaluation criteria when evaluating similar units.
4 A considerable dissimilarity between research objects can, in my opinion, lead to a situation where the comparative evaluation is less concrete and the analysis of the research results stays on the level of mere description. In other words, when evaluating structurally, functionally, or culturally very different systems or organizations one has to operate with such unspecific concepts and criteria that the analysis of the research results produces (in the sense of evaluation) secondary and perhaps even useless information. I support my argument with Etzioni-Halevy’s (1990: 1198-120) observations on the importance of the level of comparison. The level of comparison means, as already mentioned, the entity chosen for the comparison. During the research process this entity will be divided into comparable “slices”. This division often involves the problem of the macro- versus micro-comparisons. That is, at the basis of the micro level comparisons it is/may be difficult to draw any conclusions in the macro level and vice versa.
2. Categories of evaluation selected to the closer examination 2.1.
Outcome and goal attainment evaluations
Here the first main category of evaluation research is outcome and goal attainment evaluation, which has to be considered one of the most essential forms of evaluation research through the ages. The reason for putting outcomes and goals under the same title is that these two aspects are so closely related to each other that they here can be analysed simultaneously. The definition of organizational goals by Murphy Cleveland (1995: 80-82) also supports my interpretation. In their definition these researchers claim that: “organizational goals have two aspects, intent and outcomes. The former could be seen as the process that makes goal attainment possible…The second aspect refers to a criterion for organizational effectiveness (i.e., has the organization attained or accomplished goals?”). The superiority of outcome and goal based evaluation is described by Shadish, Cook, and Leviton (1991: 227-228) and Weiss and Rein (1991: 143). They discuss very thoroughly the idea that the function of organizations and projects has been planned to accomplish some specific goals and outcomes. Therefore one can evaluate the success of particular organizations by analysing the relation between the function and the accomplished goals. One of the most typical assumptions in the goal-based evaluation is that the functions of the organization can be measured and so the evaluation is seen as a measuring process which utilizes different standards, indicators, and measures. In other words, one assumes that there is always at least one indicator with which one can analyse the goals and the results already accomplished. In the goal-based evaluation the objects of the evaluation are goals and targets, and the measuring process describes their attainment degree and also sets their value. The goal-based evaluation processes use such common concepts as inputs and outputs. In the
5 evaluating situation inputs are expressed primarily in monetary terms. Also outputs are regarded in a monetary value, if the nature of the outputs allows it. Goal-based evaluation is normative in nature, in other words it tries to specify the things which an organization has to do in order to become effective. This means that the organization has to have an idea about the situation it wants to achieve and the attainment of which it wants to examine with the help of evaluation. This normative nature also involves a certain principle of causality. One believes, for example, that the function of an organization decides whether the goals will be achieved or not. From the point of view of evaluation this can mean that the viewpoint of the evaluation should be directed in a way which provides information to the leaders of the organization so that they can make the organization´s function more effectively. The problem with the normative evaluation is that it is difficult to show in a rational and empirical manner why, and on what grounds, the selected criteria would represent the true effectiveness of the organization. The normative evaluation model can be considered equal to summative evaluation, since in summative evaluation the emphasis lies on the evaluation of the results gained by the function. Characteristic of the summative evaluation is in ex post –type of evaluation, in which the function of an organization will be evaluated afterwards by counting, i.e. summing up the results the organization has gained. From the point of view of methodology the problem with the summative evaluation is that the manner of approach does not include any discussion on the reasons for the results. The question why the evaluated function has the observed results remains unanswered. (Chambers et.al. 1992: 8, Stake 1991: 303). Goal attainment evaluation emphasizes the functional features of organizations. This method strongly focuses on function, but in a way which does not pay much attention to the implementation processes themselves, but merely to the result of the implementation. Because of this the goal-based evaluation includes - at least implicitly - an idea of comparability between the implementation and the effectiveness of the organization. For this reason the goal-based evaluation can be considered a type of evaluation which emphasizes comparison. Thus the essential question is: on which level will the comparison be performed in the goal-based evaluation? Very often in the goal attainment evaluation there is a comparison between the goal attainment degrees of for example different parts of the organization or different projects. In these cases the comparison can, in my opinion, be on two levels. First of all, one can compare the accomplished outcomes with the set goals. Another possibility is to compare for example the attainment degrees of different organization units with each other. In this way it is possible to conclude how efficient the different units have been. When forming the previously mentioned situation for comparison, one has to consider a challenging question of principle, namely whether the comparison and the analysis of the accomplishment of the attainment degree will be performed with relative or absolute values. Unfortunately it seems that these
6 situations for comparison have usually been built on absolute values alone. To proceed in this way means that one is not able or willing to include any discussion in the analysis of the evaluation results on for example whether the goals are realistic regarding to the resources available. Yet another possibility to apply the comparison aspect of the goal-based evaluation is the time series–comparison. In practise this means that the results achieved by the organization will be examined as certain cycles and there will be conclusions on how the results of the organization have developed. Time series –analysis is most likely to be used when examining the organization’s internal structure but in principle it is also possible to use it to compare the attainment degree between different organizations. 2.2.
Cost oriented evaluations
Another main category of the evaluation research is evaluation which focuses on economy. As an example of this type I will use cost-benefit and cost-effectiveness analysis. All research which belongs into this group is significant, since one of the most essential areas of economic evaluation has always been comparison between the expenses and the benefits and effects gained. The essential hypothesis in the cost-benefit analysis is that the inputs and outputs used in the implementation have a direct connection with each other (the ratio of inputs to outputs) and that all costs and benefits involved in the activity can be included in the analysis. In addition, it is assumed that the ratio of inputs and outputs indicates the optimal usage of resources. In the analysis the inputs and outputs used in the function will be examined in a monetary value. (Sintonen et.al 1997: 206-211) With the help of the cost-benefit analysis the function of organizations will be evaluated by proportioning the costs to the benefits. Comparison is included as a starting point in this method. It allows three different comparative evaluation possibilities. First of all, one can compare all the desired effects (benefits) with related sacrifices (costs) in order to judge whether the desired function is worth implementing or not. Secondly, with cost-benefit analysis one can decide which alternatives are most useful. When the ratio of costs and benefits of one alternative appears to be higher than one, this alternative can be considered effective. This applies even to the alternative which has a positive net benefit, i.e. the ratio of costs and benefits is higher than zero. And finally, one can decide which collection of functions or projects constitute the best expenditure within the budget limit. (Chambers et.al. 1992: 238-240) From the point of view of the comparative evaluation the application of cost-benefit analysis requires that all criteria and measurement devices are commensurable. This
7 enables the processing of the information. The requirement may lead to conceptual problems, especially the costs can be difficult to make operational. Cost-effectiveness analysis is methodologically close to the cost-benefit analysis, with the difference that in the cost-effectiveness analysis the question is “what kind of benefits the function produces” and not “what are the costs of the function”. In the same way as in cost-benefit analysis, the cost-effectiveness analysis examines the inputs and outputs of the activity. This is usually expressed as costs per an attained result. To be able to apply the cost-effectiveness analysis, one has to identify from the organization’s function the effects produced by the function and the costs required in attaining them. It is significant that there is no attempt to express the effects in monetary value but mainly in other quantitative items of effect and benefit. (Rossi & Freeman 1985: 14, Sintonen et.al. 1997). Rossi and Freeman (1985: 330) emphasize the comparison aspect connected with the evaluation of effects. According to them, the essential principle in the evaluation of effectiveness is to analyse the relation between certain phenomena. In practice this means that a descriptive account of the attained goals cannot be considered as an evaluation of effects. In a comparison based on the cost-effectiveness analysis the criteria used in describing the costs is proportioned to the effects accomplished with the action. This means that the comparison, in my mind, should be directed to evaluate the accomplishment of the effects aimed for. The positive characteristics of the cost-benefit and cost-effectiveness evaluation as a comparative method are, from the viewpoint of economic evaluation, connected with the high standard of comparison. Especially in the cost-benefit analysis the requirement for monetary values for costs and benefits makes it possible to apply the comparison method, since monetary units can be used to measure both quantities. It is also worth noticing Levin’s (1983: 18) assumption that only (program) alternatives with similar or identical goals can be compared. There is nevertheless some critical points in using cost-benefit and cost-effectiveness analysis as a device for comparative evaluation. For example the measurement of the functions of public welfare organizations by using merely quantitative criteria does not give a whole picture of that organization´s functions. (Vartiainen 1994). It is also important to notice that especially the cost-effectiveness analysis cannot be used to compare the effectiveness of the functions of different organizations, since the effects of the functions cannot be evaluated with the same criteria in different organizations. With this method, it is possible to compare only the relative effectiveness of the function, in other words the costs with which certain effects are possible to achieve in each case. The interesting article of House (2000) is referred to as a more thorough critical discussion on the limits of the cost-benefit analysis.
2.3.
Implementation studies
8 Chambers et.al (1992: 8-9) define the implementation studies as a third main type of evaluation research. Its generality, especially in evaluations of implementation processes, supports the fact that implementation studies can be analysed as an independent type of evaluation. One example of an evaluation method which can be identified with implementation studies is empowerment evaluation. Implementation studies are a part of the evaluation of public function and politics. For this reason the evaluative implementation studies apply concepts which are typical for the policy analysis. In this article I do not have the possibility for a more thorough discussion on the definitions of policy analysis and implementation evaluation. Instead, I refer to e.g. Leeuw’s (1995: 19-23) interesting article, in which these questions are widely discussed. The difference between goal-based evaluation and evaluation which applies implementation studies is that in the latter execution is examined as a process entity, although in many parts the focus is on the goals. The evaluation research based on implementation studies focuses primarily on problems. It tries to discover problems which can appear during implementation processes, since it has been noticed that a failure during the execution can prohibit the goal attainment even though the conditions for the execution would be in order. Berndtson (1992: 261-262) and Premfors (1989: 16) both emphasize the problemfocused nature of the policy-analysis research, consequently defining the rest of the evaluation research in social sciences as economic and macro political. According to them this tendency can be observed in the way evaluation research in social science focuses on the cost-benefit and cost-effectiveness analyses, whereas the more practical policy-analysis always has its starting point in solving the problems. Yet to be exact one has to remind that the focus on the economy was typical in the evaluation research in social sciences during the 1960’s and 1970’s, although it gained wider dimensions already at the end of 1970’s. In the administrative sciences, for example, the evaluation research of the public sector implementation have in the last decade been mostly qualitative and have focused on the problems of the implementation. Empowerment evaluation is defined as a method which helps (program) participants and employees to evaluate themselves and their implementation in order to improve practice and to foster self-determination. (Fetterman et.al. 1996: 4) The most interesting point in the empowerment evaluation is that the participants conduct their own evaluations and usually act as facilitators. This method aims to increase the rights of self-determination by using different research methods. The role of the researcher is to teach the target organization the principles of evaluation. Consequently, the role of the evaluator is to act as a trainer and a teacher in the evaluation process, not as an objective researcher. The most important point in the empowerment evaluation is that its aim is not merely to evaluate the quality of the implementation but more importantly to develop the existing execution process of the organization or program.
9
Empowerment evaluation has its roots in community psychology. Community psychology focuses on people, organizations, and communities working to establish a control over their own affairs. Therefore, the literature on citizen participation and community development can be seen as a part of empowerment evaluation’s development. On the other hand, empowerment evaluation also derives from collaborative and participatory evaluation. (e.g. Whyte 1990) One of the goals in the empowerment evaluation is to decrease the mystique connected with evaluation and to make it easier for the organizations to start more versatile evaluation processes and practises. This goal is supposed to be achieved with the help of five theses, which are as follows: training, facilitation, advocacy, illumination, and liberation. (Fetterman et.al. 1996: 9-11) The process of the empowerment evaluation consists of four stages or steps. The aim of the first step, taking the stock, is to chart the present situation of the evaluation object by collecting all the essential factors connected with the activity. On the grounds of these factors one can “create a base line from which future progress can be measured”. The second stage of the empowerment evaluation is setting the goals. What is essential in this part of the evaluation process is that the setting of goals is proportioned to the present condition of the evaluation object. In this way, the goals will show the direction in which the function should go in the future. In other words, the goals are proportioned to the activity. The third stage in the empowerment evaluation process is developing the strategies. Program participants are responsible for selecting and developing strategies to accomplish program objectives. In this work one has to be aware that the two previous steps will not be ignored in the strategies. The fourth stage is documenting the progress. All those involved in the evaluation process have an opportunity to influence the way the information produced by the function and evaluation -and the possible development- will be documented. (Fetterman et.al. 18-20) Examining the implementation studies from the point of view of comparative evaluation one can find that the comparison between execution processes and their evaluation results can produce problems due to the differences between the processes of different units. In my opinion, there is no point in comparing them from the point of view of empowerment evaluation, since its method includes a specific target, i.e. empowerment evaluation depends on the evaluation object, the need for the evaluation, and the person carrying out the evaluation. If implementation studies (in a wider sense) are being used as a method for comparative evaluation, the research situation demands that at least the background and the practises of the processes being evaluated will be thoroughly charted. In this way one can form a justified picture of the comparability of the analysis.
2.4.
Consumer satisfaction study
10 Evaluation research describing the satisfaction of customers and users of different services is nowadays very common. In this type of research the emphasis is on the aspects which describe first of all the quality of the projects and the services produced, as well as the customers’ opinions of the function of the evaluation object. As an example of this type of evaluation method I will use the servqual-method which analyses the quality of services. The evaluation of service quality includes mainly evaluation and measurement of abstract objects. Servqual-method has been developed to measure the service quality as the customers’ satisfaction on the services they receive. This method is well known in the TQM system. An essential thesis in servqual is the assumption that the quality of the services culminates partly on the customers’ expectations and partly on their perceptions. It is assumed that there is a so called service gap between these two states, the existence of which is supported by the difference between the measured values of customer’s expectations and perceptions. The development of this method started with an observation which shows that the customers evaluate quality with same principles regardless of what kind of service is referred to. As a result of extensive empirical research, ten quality aspects (describing all services) were found. Further development of these aspects brought five final quality dimensions. Measuring devices for the servqual were based on these dimensions as well. (Parasuraman-Zeithaml-Berry 1988) The method contains five dimensions which have been divided into 22 statements. Each of these statements will be evaluated on a scale of 1-7 in which seven presents the best possible quality. Individual customers may not, however, consider these dimensions of equal importance. By dividing 100 points among these dimensions they can express their own preferences. The research method can therefore be considered sensitive towards the individual opinions of the customers. According to the same principle the research results can be emphasized according to the views of different individuals. In this method the research results will assume a concrete form as so called servqual-points. This is achieved by subtracting the points of “the experienced service” from the points of “the expected service”. In this way one can find out the customers’ individual estimations on how satisfied they are with the quality of the service they have received. The analysis of the results also gives valuable information for management: the possible service gaps will be discovered and measures can be taken to prevent them. In my opinion, servqual can be used as a method for comparative evaluation if it is done carefully. With servqual one can compare the defects in the customer satisfaction in different parts of the organization and to discuss the possible reasons for these defects and problems. Servqual is probably most efficient as an evaluation method for organization’s internal affairs. In the comparative sense it should be used mainly in a time series-analysis in which the development of the customer satisfaction will be evaluated with repeated surveys.
11
2.5.
Needs studies
Need study has been chosen as the fifth category of evaluation research. The aim of the need study –both at the planning as well as the starting stage- is to evaluate the necessity of the activity in question. Need studies are also applied to evaluate how well the goals set for the activity meet the customers’ needs. The term need has several meanings. The activity of the public service organizations derives from the assumption that the services and their production are based on the clients’ needs in different sections of the welfare policy. Conceptually the term need has been defined in many different ways. From the point of view of this article the most interesting one appears to be Bradshaw’s (1977) typology, in which he divides the needs into four main categories. The first category consists of normative needs, which are defined by comparing the conditions of individuals and groups with the accepted standards. For example the subsistence level is one such standard. If a person earns less than the standard he/she is considered to be in the need of financial support. The second category consists of felt needs which are based on the individual’s own experiences about his/her needs. In other words, felt needs are based on the perception of an individual experiencing a need. Felt needs can be identified with desires. They can derive from a genuine need, especially if the individual is not capable of asking for help in satisfying his/her needs. The third category, expressed needs, consists of the attempt to satisfy the felt needs. Expressed needs can be identified with the demands that individuals make to the service industry. The fourth category consists of comparative needs which are defined with the help of reference groups. This means that those individuals or groups who receive services are compared with those who do not use the service in question. Another possibility is to examine the needs and their extent by comparing the circumstances of the service users for example in the rural and in the urban areas. (Chambers et.al. 1992: 68-69) How to define evaluation focused on needs? It is essential to bear in mind that needs studies are directed to the responses of the expectations which exist between the service industry and its users. Rossi and Freeman (1985: 104) give a similar definition, even though they emphasize the problems connected with the responses. They define the need-based evaluation as follows: “a systematic appraisal of type, depth, and scope of problems as perceived by study targets or their advocates”. Thus the focus of the need-based evaluation is on the orientation to the function of the service organization and on the problems observed in it. Naturally, the need-based evaluation can also be defined directly from the concept of need, which was presented above. This is what for example Mayer (1985: 127) does when he states that the need-based evaluation can be defined as “the difference between the extent of a
12 condition or need in a given population and the amount of services provided to meet that need”. In the evaluative need study the examination can be performed at all the levels of the social life. Many studies of similar nature are performed at the customer level, but evaluation can also be done at the organizational, communal, and social levels. Very similar methods are being used in evaluating these levels. There are various methods for performing a need study. One example could be the focus group method. The method of focus group interview can be applied in the evaluative need study for example by interviewing users of those particular services which are being evaluated. With the help of a group interview the researchers try to obtain profound information on services and the needs directed to them by creating an interactive situation with the service users. In the central part of the focus group method is a group selected to an interview. The intention of this interview is to produce qualitative information about service needs by discussing the subject. Krueger (1988: 27) describes the focus group method and the differences between this method and other similar methods by emphasizing the composition of the focus group: “(a) people, who (b) posses certain characteristics, (c) provide data (d) of a qualitative nature (e) in a focused discussion”. One or more corresponding elements can appear in other group methods, but not in the same context as in the focus group method. With the help of the focus group method one can produce evaluative research data on needs connected to the function before, during, and after it.
3. Conclusions It is possible to apply evaluation methods to the process of the comparative evaluation research, and for some of these methods comparisons are even natural. To connect comparison with the evaluation process is natural for example in the goalbased and economy-based evaluations. This does not, however, mean that I would consider comparative evaluation applicable to all evaluation situations, processes, and methods. I will discuss the matter in the light of Table 2, in which I have collected the characteristics of different evaluation methods according to the four viewpoints examined in the article. It should be noted that the division of the characteristics into different viewpoints is not always unambigious. Especially the division between the two principles “the object of evaluation” and “the level of comparative evaluation” is in some parts open to various interpretations. Table 2. A summary of the characteristics of comparative evaluation Principle
Selection of the evaluation object
The level of comparative evaluation
The conceptual comprehension
The analysis of evaluation research
13 Evaluation category Outcome and goal attainment evaluations
results organizations, programs
The degree of goal attainment organization´s internal time series-analyses
Cost oriented evaluations cost-benefit analysis
organizations, programs, aternatives for implementation
Implementation studies empowerment evaluation
the optimal usage of the resources
organizations, programs, aternatives for implementation
is inapplicable in comparison between organizations
execution, implementation alternatives
process entity
quantitative causal normative summative
established: costs and benefits in monetary value
searching for the most applicable alternative price comparison
relations between phenomena
or
cost-effectiveness analysis
the comparison between alternatives
Established: inputs, outputs, goals
comparison of net benefits
costs, effects and benefit items
comparison of benefits and their differences examination of the efficiency of the effects
is formed during the evaluation process
analysing the problems and their solutions analysing the conditions for the function
Customer satisfaction study servqualmethod
customers
individuals, groups
Established: tangibles, reliability, responsiveness, assurance, emphaty
the quality of services: comparing customers expectations and
14 perceptions comparing service gaps Needs studies focus groupevaluation
the stage for planning and starting the function, reform situation
individual, community, society
are formed within a group situation
comparing the customers´ needs and the service production
As already mentioned there is a comparative aspect connected with the goal-based evaluation, which is emphasized in the comparison of the effectiveness of different organizations or different parts within an organization. This view is supported by the assumption of the quantitative measurement of the function connected with the method. When applying the method to the comparison between different organizations one has to pay attention to the operating of the concepts. The goalbased evaluation uses established concepts, however operating these concepts from case to case can produce problems of comparability. The assumption that one can set a monetary value on the inputs and outputs makes it possible to apply the economy-based evaluation to the comparative evaluation process. Even in this method the main concepts are established but the definition of the context of the benefits and effects requires target specific discussion. Especially from the point of view of the cost-effectiveness analysis it can be difficult to find criteria which will analyse the effects of different organizations’ functions comparatively. It should be noted that the comparative aspect of the economy-based evaluation is the most plausible one in examining the alternatives for potential functions. Both the implementation studies and the customer satisfaction studies are, in my opinion, relatively problematic, as tools for comparative evaluation since evaluation in these methods is directed to case specific and subjective elements. The quite applicable tool for comparative evaluation is servqual, which was mentioned in the article. It is a method which is far standardized and uses concepts which are not dependent on the target. Need studies as a method for comparative evaluation is supported by the fact that it emphasizes the situation of planning and reform. In my opinion, the need-based evaluation can be used as a comparative evaluation process in charting the customers needs and the changes in them. This method can cause problems because of its unestablished and case specific concepts. Therefore these concepts should, if applied to the comparative evaluation, be define so that the comparison of the research process will be possible.
15
REFERENCES Argyris, Chris (1962). Interpersonal Competence and Organizational Effectiveness. Richard D. Irwin, Inc. Homewood, Illinois. Berndtson, Erik (1992). Politiikka tieteenä. Johdatus valtio-opilliseen ajatteluun. Valtionhallinnon kehittämiskeskus. Painatuskeskus, Helsinki. Bradshaw, J. (1977). The concept of social need. Teoksessa: Planning for social welfare: Issues, models, and tasks, 290-296. Toim. Gilbert, N. – H. Specht. Englewood Cliffs, NJ: Prentice-Hall. Chambers, Donald E., Kenneth R. Wedel & Mary K. Rodwell (1992). Evaluating Social Programs. Boston jne.: Allyn and Bacon. Cheetham, J., E.J. Mullen, H.Soydan & K. Tengvald (1998). Evaluation as a Tool in the Development of Social Work Discourse. National Diversity or Shared Preoccupations? Reflections From a Conference. Evaluation 4(1): 9-24. Etzioni-Halevy, E. (1990). Comparing Semi-corruption among Parliamentarians in Britain and Australia. Teoksessa: E. Øyen (ed.), Comparative Methodology: Theory and Practice in International Social Research, SAGE Publications Ltd. London, 113-133 Fetterman, David (1996). Empowerment Evaluation: An Introduction to Theory and Practice. Teoksessa Empowerment Evaluation. Toim. Fetterman, David & Shakeh, Kaftarian & Abraham, Wandersman, 3-46. Thousand Oaks: Sage Piblications. House, Ernest R. (2000). The Limits of Cost Benefit Evaluation. Evaluation 6:1; 79-86.
16
Krueger, Richard A. (1988). Focus Groups. A Practical Guide for Applied Research. Beverly Hills, California. SAGE. Leeuw, F.L. (1995). Policy Theories, Knowledge Utilizations, and evaluation. Teoksessa Policy Evaluation. Linking Theory to Practice Toim. Ray C. Rist, 20-37. Aldershot: Edward Elgard Publishing. Levin, H.M. (1987). Cost-benefit and cost-effectiveness analysis. In D.S. Cordray, H.S. Bloom and R.J. Light (Eds.), Evaluation practice in review, 83-99. San Francisco, CA:Jossey-Bass. Manns, C.L & March, J.G. (1978). Financial adversity, internal competition, and curriculum change in a university. Administrative Science Quarterly, 23: December, 541-542. Mayer, Robert R. (1985). Policy abd program Planning. A developmental perspective. Englewood Cliffs, Prentice-Hall, Inc. Murphy, K.R. – J.N. Cleveland (1995). Understanding Performance Appraisal. Social, Organizational, and Goal-Based Persoectives. Sage, Thousand Oaks. Parasuraman, A - Zeithaml, Valerie A. - Berry, L. (1988) Servqual: A Multiple-Item Scale for Measuring Consumer Perceptions of Service Quality. Journal of Retailing, 64: 12-40. Pennings, J.M. – P.S. Goodman (1979). Toward a workable framework, teoksessa New Perspectives on Organizational Effectiveness, ed. J.M. Penings – P.S. Goodman, 146-148. Jossey-Bass, San Francisco. Peters, B.G (1988). Comparing Public Bureaucracies: Problems of Theory and Method. Tuscaloosa: Univeresity of Alabama Press. Peters, B.G (1998). Comparative Politics: Theory and Method. Basingstoke: Macmillan. Premfors, Rune (1989). Policyanalys. Kunskap, praktik och etik i offentlig verksamhet. Studentlitteratur, Lund. Ragin, C.C. (1989). The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies. Berkeley: University of California Press. Rose, R (1991). Comparing Forms of Comparative Analysis. Political Studies XXXIX, 446-462. Rossi, Peter H. – Howard E. Freeman (1985). Evaluation A Systematic Approach. Beverly Hills, Clifornia. SAGE. Shadish, William R., Thomas D. Cook & Laura C. Leviton (1991). Foundations of Program Evaluation: Theories of Practice. Newbury Park: Sage Publications. Sintonen, Harri – Markku Pekurinen – Eero Linnakko (1997). Terveystaloustiede. Helsinki, WSOY. Stake, Robert E. (1991). Program Evaluation, Particularly Responsive Evaluation. Teoksessa: Evaluation Models: Viewpoints on Educational and Human Services Evaluation, 287-310. Toim. George T. Madaus & Michael Scriven & Daniel L. Stufflebeam. Boston jne.: KluwerNijhoff Publishing.
17 Vartiainen, Pirkko (1994). Palveluorganisaatioiden tuloksellisuusarviointi, teoriat ja käytäntö. Päivähoitoon, erityisesti päiväkoteihin kohdistuva monitahoarviointi. Acta Wasaensia 40, hallintotiede 2. Vaasa: Vaasan yliopisto. Weiss, C.H. (1972). Evaluation research: Methods for assessing program effectiveness. Englewood Cliffs, NJ: Prentice Hall. Weiss, Robert S. & Martin Rein (1991). The Evaluation of Broad-Aim Programs: Experimental Design, Its Difficulties, and an Alternative. Teoksessa: Evaluation Models: Viewpoints on Educational and Human Services Evaluation, 143-161. Toim. George T. Madaus, Michael Scriven & Daniel L. Stufflebeam. Boston jne.: Kluwer-Nijhoff Publishing. Whyte, W.F. (1990). (Ed.). Participatory action research. Newbury Park, CA: Sage. Zelditch, M. Jr. (1971). Intelligible Comprarisons. In Sociology: Essays on Trends and Applications, 267-308. Ed. Ivan Vallier. Berkley etc.: University of California Press.
