USE OF MULTICRITERIA METHODS FOR THE ASSESSMENT OF TRANSPORT PROJECTS Tsamboulas1 D., Yiotis2 G.S., Panou3 K.D.
ABSTRACT This paper attempts to consider some key elements of the transport assessment process in order to present and compare the most commonly applied multicriteria methods to be used in the assessment of transport projects. This comparative analysis is based on the performance of the methods with respect to transparency, simplicity, robustness and accountability. To assist the comparison of the different methods, an application example consisting of three transport projects in Greece is used. Finally, the methods’ advantages and disadvantages are systematically presented so as to allow for the decision-maker to draw definite conclusions on the appropriateness of each method for the different fields of transport appraisal.
INTRODUCTION Scheme and project evaluation has become an important component of modern transport planning and administration. In transport appraisal practice there are several methods used for the evaluation of the various transport initiatives (the term “initiative” can be used for a project, an investment, or a policy). These methods are usually grouped in two major categories: the single-criterion (monetary approach), and the multicriteria methods (nonmonetary approach). While the former reduces all possible impacts to one single monetary criterion, the latter uses more than one criterion, not necessarily quantifiable. Traditionally, a monetary approach has been used for the evaluation of transportation systems. This has been done by using a cost-benefit analysis to account for the monetary costs and benefits that were usually directly related to the transportation system under consideration. Systems’ related costs were often estimated in terms of capital cost,
1
Civil Engineer, Assistant Professor at the Faculty of Civil Engineering - Dept. of Transportation Planning and Engineering, National Technical University of Athens, 5,Iroon Polytechniou St. GR-157 73 Zografou, Athens. Tel: +30 1 7721367, Fax: +30 1 7721327, Email:
[email protected] 2
Civil Engineer, Researcher at the Faculty of Civil Engineering - Dept. of Transportation Planning and Engineering, National Technical University of Athens, 5,Iroon Polytechniou St. GR-157 73 Zografou, Athens. 3
Researcher, National Technical University of Athens, 5,Iroon Polytechniou St. GR-157 73 Zografou, Athens.
operating costs and return on investment. However, when safety and environment became important issues, safety, pollution and travel comfort related costs were also included. Over the years monetary evaluation approaches have aroused some criticism. Such critical remarks stem mainly from attempts to put the underlying welfare economic theory into practice. There are often considerable difficulties in measuring all relevant impacts of a project or plan in money terms. Although many efforts have been made to produce values for intangibles and/or externalities (Pearce, 1978), it is almost impossible in practice to arrive at totally reliable and fully accepted monetary values for such impacts. Furthermore, political priorities cannot be taken into explicit account in such a monetary evaluation. Due to these important limitations, monetary evaluation has been complemented in the past decade with a variety of non-monetary evaluation methods known as multicriteria methods. It is noteworthy that the debate on conventional Cost Benefit Analysis (CBA) and multicriteria analysis tends to regard these two approaches as complementary rather than competitive analytical tools. The main task of the present paper is to emphasise the non-monetary evaluation tools and test the appropriateness of selected multicriteria methods for the assessment of transport infrastructure projects. This, by no means, implies that other approaches not considered here (such as the EVAMIX, the ORESTE, the QUALIFLEX, the UTA, the PROMETHEE method, etc.) should be discarded as unsuitable (Willis, 1980).
PRESENTATION OF THE MULTICRITERIA METHODS Transport evaluation is essentially conflict analysis characterised by technical, socio-economic, environmental and political value judgements. Therefore, in a transportation planning process it is very difficult to arrive at straightforward and unambiguous solutions. This implies that such a multi-related planning process will always be characterised by the search for acceptable compromise solutions, an activity that requires an adequate evaluation methodology. Multicriteria evaluation techniques aim at providing such a set of tools as much as providing a flexible way of dealing with qualitative multidimensional effects of transport initiatives. This however, does not mean that multicriteria evaluation is a panacea and can be used in all circumstances without difficulties; it has its own problems. In general, the main shortcomings related with multicriteria models stem from the following methodological assumptions:
2
1.
There is no solution optimising all the criteria at the same time and therefore the decision-maker has to find compromise solutions. Thus, when different conflicting evaluation criteria are taken into account, a multicriteria problem is mathematically ill structured.
2.
The relations of preference and indifference are not enough in this approach, because when an action is better than another for some criteria, it is usually worse for others, so that many pairs of actions remain incomparable with respect to a dominance relation. A great number of multicriteria methods have been developed and applied for different appraisal objectives in
different contexts (Bana e Costa, 1990). Although most of these methods were designed to cover a wide variety of problems, in practice they can be applicable and provide effective solutions only in particular decision situations. Therefore, it can be thought that not all the multicriteria methods are capable of meeting the standards and addressing the particularities of transport evaluation; and that the identification of the most suitable methods is of great importance.
Identification of Methods In order to perform a pre-screening between candidate methods for transport evaluation, a matrix of applicable methods versus possible decision situations was initially established.
This required on one hand the review of
representative MultiCriteria Analysis (MCA) methods which have a track record of application and user acceptance in the appraisal practice, and on the other, the examination of their applicability, data requirements, ease of use and utility of results, to a diverse set of problem situations. Decision situations were differentiated by level (e.g., strategic, local), type of project, availability/accuracy of data, etc. Based on such a matching of methods versus problems, the most suitable methods were identified: 1. REGIME
4.
2. ELECTRE Family 3. AHP
Multi-Attribute Utility Approach
5.
ADAM type
Table 1 shows how the selected methods were grouped according to the conventional decision-theory typology, and the formalistic approach using their mathematical structure. TABLE 1
3
Before investigating in detail the pros and cons of the selected methods and their overall suitability for transport evaluation, the methodological principles and philosophical characteristics of these methods will be briefly presented. This will allow for the potential transport planner to gain a clear understanding of the factors underlying the interface between the theoretical acceptances and their practical justification.
The Vectorial Model These methods are based on the assumption that all feasible solutions of a decision problem can be represented as vectors in a vectorial space of dimension equal to the number of the evaluation criteria. Dividing the total solution space to subspaces (where projects can be ranked with certainty) and comparing them according to their relative size leads to the assessment of decision alternatives (e.g. projects). This approach may prove quite helpful when only qualitative information is available, or lead to a considerable loss of information when quantitative data also exists.
REGIME (Hinloopen et al. 1983, Nijkamp et al. 1993) An example of a multicriteria method using a vectorial model is REGIME. This method uses pairwise comparisons and from this point of view it can be considered as an outranking method. However, it is based on a linear additive model and thus, it can also be considered as of this type. The method’s point of departure is an ordinal evaluation matrix and an ordinal weight vector. In regime analysis the main focus is the sign of differences between impacts of alternatives. By calculating these signs for each pair of alternatives (and for each criterion) the regime matrix R is generated. This matrix is simply a transformation of the impact matrix. The sign can be determined with certainty for certain regimes. For others however, no such unambiguous result is possible. Such a regime is called a critical regime. Critical regimes are covered with uncertainty, since no conclusion can be drawn about the sign of the differences between impacts. The main idea of regime analysis is to circumvent this uncertainty by partitioning the set of feasible weights so that for each subset a definite conclusion can be drawn about the sign.
The Superiority Graph Model.
4
The base of the so-called “superiority graph models” is the concept of partial comparability. This concept stems out of the understanding that in multicriteria problems the dominance relation can be supported only if there exists a consensus of points of view. Thus, an action a would outrank an action b only if a is at least as “good” as b on all criteria considered. The "superiority graph models" are treating preferences as ordered outranking relations. To do so they are using vectors. Each vector is directed from the superior to the inferior alternative, for each alternatives’ pair, when certain presumptions exist (i.e., when an outranking relation can be established). The innovative part of this approach is that it uses the notions of "Concordance” and “Discordance". While the former implies the degree of dominance of an alternative over another, the latter implies the degree of inferiority of an alternative to another. The most widely acceptable methods in this category are the ELECTRE family (Roy 1985, Schaerlig 1985, Szidarovsky 1986).
ELECTRE (Roy 1985, Schaerlig 1985, Szidarovsky 1986). In ELECTRE the decision-maker primarily determines the Concordance and Discordance limits. Following this stage, all pairs of alternatives are tested for valid outranking relations and the positive and negative flows are computed (an axiomatic characterisation can be found in Bouyssou and Perny, 1990). Final rankings are determined on the basis of the intersection of these flows (ELECTRE I). ELECTRE II follows more or less the same process of implementation, with a small variation in the computation of the Concordance index. As mentioned before, the concept behind the superiority graph methods is that the dominance relation can be enriched only if realistic information is available; therefore, there is a formal structure between the dominance relation which is too “weak” and a complete preorder. From the ELECTRE family the only method that addresses the weak preference issue is the ELECTRE III. In this method, the term “weak preference” refers to the case where alternative i outranks i΄ (for a given criterion j), but not to that extend so as to result in clear dominance. In ELECTRE III, acceptable outranking relations must also satisfy the veto threshold constraints.
5
In ELECTRE IV, the Concordance and Discordance indicators are not used. Instead, the determination of some proportionality functions is required to denote the outranking relations between pairs of alternatives. As the majority of the methods using the partial comparability concept, ELECTRE IV also takes into account the intensity of preference of an alternative solution over another. Thus, it uses the notion of “strong” and “weak” dominance, which is implemented by considering appropriately selected veto thresholds.
The Additive Model This approach’s primary objective is to establish a “performance norm” on the basis of which all the alternatives are compared. Performance norms are usually linear. The main assumption behind their definition is that the decision-maker's preferences are a complete preorder i.e., all decision alternatives are comparable and transitivity of preference and indifference relations holds. Many methods have been proposed using a simple additive model. These methods usually differentiate on the basis of implementation of the additive function. According to Munda (1995), there are three different schools in the current practice: the "Multiple Attribute Utility" (Beuthe 1996, Schaerlig 1985), the “Analytical Hierarchy Process: AHP” (Saaty 1980) and the "Ideal Point approach” (Zeleny 1982).
Multiple Attribute Utility Approach (Schaerlig 1985). This approach is based on the following hypothesis: in any decision problem there exists a real valued function U defined on the set A of feasible alternatives, which the decision maker wishes, consciously or not, to examine. This function aggregates the different criteria taken into account, so that the problem can be formulated as: max U(gj(a)) : aA where U(gj(a)) is a utility function aggregating the n criteria (therefore a multicriteria problem is reduced to a monocriterion one). The role of the analyst is to determine this function. The most commonly used functions are linear or multiplicative. However, in some cases, a utility function may be crooked linear (Beuthe 1996), with partial sections consisting of straight-line segments.
6
Analytical Hierarchy Process (AHP) (Saaty 1980). The Analytic Hierarchy Process is perhaps the most commonly used method for the prioritisation of decision alternatives, having a wide variety of possible applications. The method is a systematic procedure that organises the basic rationality of a decision problem by breaking it down into smaller constituent parts and then calls for only simple pairwise comparison judgements to develop priorities in each hierarchy. There are three principles that AHP recognises in a decision situation, namely the principles of decomposition, comparative judgements and synthesis of priorities. The decomposition principle calls for structuring the hierarchy to capture the basic elements of the problem. This is done by either working downward (from the focus in the first level to criteria bearing on the focus on the second level, followed by sub-criteria in the third level, and so on, from the more general to the more particular and definite), or working upward starting at the bottom, identifying alternatives for that level and attributes under which they should be compared, which fall in the next level up, and so on. The principle of comparative judgements call for setting up a matrix (aij) to carry out pairwise comparisons of the relative importance of the elements in the second level with respect to the overall objective (or focus) of the first level. Comparisons are made judgementally by the decision maker using the 1-9 scale (1 for equal importance, 9 for extreme importance of one alternative over the other). The synthesis of priorities principle is then applied. Priorities are synthesised from the second level down by multiplying local priorities by the priority of their corresponding criterion in the level above, and adding them for each element in the level according to the criteria it affects. In the AHP literature, a wide variety of ways are found to derive the vector of local priorities from the pairwise comparisons matrix (aij). As originally proposed by Saaty the most frequently used technique to obtain the priority vector is the eigenvector technique. This approach uses the largest or principal eigenvalue (λmax) of the matrices (aij) to compute the corresponding eigenvector that represents the vector of local priorities.
7
An alternative way to get an approximation to the priorities is to normalise the geometric means of the rows. This result coincides with the eigenvector for n3. Another way to obtain an approximation is by normalising the elements in each column of the judgement matrix and then averaging over each row.
The Ideal Point Approach [The ADAM method] (Zeleny 1982). The ideal point procedures are characterised by the following axiom of choice: alternatives that are closer to the ideal are preferred to those further away (figure 1). To be as close as possible to the perceived ideal, is the rationale of human choice. The main idea behind this approach is to identify the ultimately desired outcome (called the “Ideal Point I”) in the space of the n-criteria attributes of competing alternatives. In the ADAM method a depiction is made from a n-criteria evaluation problem to a n-dimension vectorial space. Then, a normalised value is found for each criterion j, on the basis of which the various alternatives are ranked first, second, etc. This value represents the distance between alternative i and the ideal point I in the vectorial space. Therefore, the smaller the distance, the more attractive alternative i can be. FIG 1.
FOCUS AND REQUIREMENTS OF THE TRANSPORT EVALUATION PROBLEM According to Banister (1997) and Mintzberg (1979) and bearing in mind the complexity and the multi-dimensional characteristics of a transport problem, a transport evaluation methodology has to present the following four main characteristics: transparency, simplicity, robustness, and accountability. Within the notion of transparency, the process must be understood and well interpreted by decision-makers. Otherwise, there is a danger that the decision process is being understood merely by a few experts.
Simplicity is also extremely important for an assessment methodology. Simplicity does not necessarily mean to trivialise reality, but rather to emphasise and clarify the pivotal points of the problem.
Robustness is another important property. In operational terms, it refers to the capability of receiving inputs concerning performance of the different transport initiatives and of generating simple outputs permitting the
8
evaluation of direct impacts as well as the assessment of indirect effects on the social and physical environment. In addition, it refers to a series of efficiency criteria such as: a) Data requirements b) Treatment of any number of projects/criteria c) Treatment of uncertainty d) Encouragement of special interests group to participate e) Sensitivity Finally, the decision-makers must feel confident that the decisions are made accountable and that they can fully agree and support them. This means that a decision should be able to be traced back through the different stages of the process. All the above constitute the main framework within which the comparative analysis of the sections that follow, was performed. To assist the comparison of the various MCA methods, an application example consisting in three transport projects in Greece was employed.
APPLICATION EXAMPLE The application example involved the assessment of three transport infrastructure investments:
the expansion of Mitilini port on the island of Lesvos (Project A1). The proposed works for the expansion of Mitilini port involved construction of breakwaters and seawalls of total length of 220m as well as shaping of 20,000m2 of land to accommodate new port facilities. The total cost of the project was estimated to rise up to approximately 4.7 million dollars in 1993 prices.
the construction of the new Patras port (Project A2). The proposed works include construction of breakwaters and seawalls, on-shore port facilities and links to existing road/rail networks and the old port. The total cost of the project is estimated to amount up to 30.5 million dollars in 1993 prices.
the construction of the Rion-Antirion fixed link (Project A3).
9
The link will be a cable-stayed bridge with a total length of 2,920m. The total cost of the project is estimated to rise up to 765 million dollars in 1996 prices. Important to note is that this project is partly financed by private sponsors (who will operate the facility for 35 years after its completion). In a following section there will be a demonstration of how the different competing multicriteria methods can tackle and solve the assessment problem for the three projects.
Structure of the Application Example An initial proposal on the impacts to be included in this example was to consider the whole list of impacts used by the authors in an earlier study for the evaluation of these projects (Tsamboulas et. al., 1998). However, the idea of using a fully detailed set of impacts was soon abandoned since it would induce a great deal of complexity, unnecessary for the illustration purposes of this demonstration example. Instead, a non-exhaustive list was used consisting of merely three impacts: 1.
maximise IRR (Internal Rate of Return), resulted from a standard cost-benefit analysis (Criterion C1),
2.
maximise safety, measured as the percentage of accident decrease (Criterion C2),
3.
minimise environmental concerns, measured judgementally on a 1-100 scale taking into account noise, air pollution, severity and landscape quality (Criterion C3). In order to determine the relative importance of the criteria a group of 15 transport experts was set up, which
came up with the following weights: IRR 40%, Safety 35% and Environment 25%. The panel of experts consisted of academics and researchers (from the National Technical University of Athens), consulting engineers (from the private sector) and policy makers, specialised in infrastructure planning and transport appraisal in Greece. The authors have requested the opinion of this group for two main reasons: (i) to implement the application example as objectively as possible, and (ii) to identify which of the competing methodologies addresses key issues of transport evaluation most effectively. In two rounds of interviews the experts have concluded the criteria weights and the alternatives’ scores against the criterion “environment”. Inputs to IRR and Safety have been derived from specific model runs (a standard CBA tool and the HIGHWAY model, which performs traffic assignment and forecasts). All scores represented changes with respect to the do-nothing scenario.
10
The impact matrix including the criteria weights is given in table 2. TABLE 2
Application of the Considered Multicriteria Methods REGIME During the method's application the total solution space S had to be subdivided into two sub spaces, K(1): representing the 3/4 of the total area of the Regime space and K(2): representing the remaining 1/4th. Following this stage, the centroids' coordinates were computed and compared against the actual weights of table 2. It has been shown that the centroid K(2) [0.44, 0.36, 0.20] offers a good approximation of the actual weights, while centroid K(1) [0.67, 0.25, 0.08] differs substantially. Based on this, the final ranking of the competing projects was determined: A1>A2>A3, and has proved to be inconsistent with the prevailing A3>A2>A1 (identified by the majority of the other methods). This can be attributed to the fact that criterion C1 has been over emphasised, as shown by the centroid's coordinates (K(1) [0.67, 0.25, 0.08]).
ELECTRE Family * Primarily, the Concordance and Discordance limits (ε and respectively) were set by the experts for the whole set of methods: ELECTRE I-II, =0.8, *=0.1 ELECTRE III, =0.8, * not used. * ELECTRE IV, instead of ε and , the “intense superiority limit” was used: p=0.4 Subsequently, the final rankings were computed on the basis of the positive (leaving) and negative (entering) flows. In ELECTRE I, only the positive flows were used that yielded: {A2, A3}>{A1}, while in ELECTRE II, the ranking A2>A3>A1 was derived on the basis of the intersection of leaving {A2}>{A1, A3} and entering flows {A3, A2}>{A1} (with the best alternative being on the left).
11
* In an attempt to explore these results further, a higher Discordance limit (ε =0.30) was tested, which produced more outranking relations i.e., A2>A1, A3>A1 and A3>A2. Thus, the final ranking became: A3>A2>A1.
MAUT, AHP and ADAM type For MAUT, two main assumptions were made: first that, for simplicity reasons, no σ(i) terms (representing the possible errors of a project’s utility estimation), should be included in the analysis, and second, that the partial utility functions Ui(φi) should be crooked linear with characteristic points:
IRR:
[(0, 0), (5, 0.20), (10, 0.30), (15, 0.40)]
Safety:
[(0, 0), (10, 0.20), (20, 0.35)]
Environment:
[(0, 0), (7.5, 0.15), (15, 0.25)]
The x co-ordinates of the characteristic points were selected by the group of experts to cover the possible field of variation of the criteria attributes, while the y co-ordinates to represent the decision-maker’s intensity of preference. Thus, a 5% increase of the IRR is preferred 0.20 (on a 01 scale, with 1 for maximum intensity), a 10% is preferred 0.30 and so on, until all criteria were defined. As concerns the AHP method, the final ranking obtained was A3>A1>A2. In comparison to the rankings from the vectorial (A1>A2>A3) and the majority of outranking models (A3>A2>A1), one may consider this particular outcome to be a “compromise solution”. This is because the eigenvector technique has yielded a weight vector ωj=[0.54, 0.30, 0.16] which can be seen as a geometric compromise of vectors [0.67, 0.25, 0.08] and [0.40, 0.35, 0.25]. A summary of rankings derived from the different methodologies is given in table 3. TABLE 3
DETAILED COMPARISON OF THE METHODS The above application example provided the basis for the examination of the methods' performance in terms of transparency, simplicity, robustness and accountability, which constitute principles that ensure the effectiveness of an appraisal methodology. This performance is as follows:
12
Transparency In order to identify the merits of the five MCA methods with respect to transparency, the experts were posed questions such as, which of the competing methods: Q1: copes better with real world situations? Q2: offers the closest to human rational approach to decision analysis? Q3: is well structured and easy to follow? Q4: uses straightforward mathematical approximations to represent reality? Their responses to these questions were aggregated and conclusions were drawn for each method. A summary of findings is presented in table 5. More specifically, REGIME analysis can represent almost any real world problem in transport evaluation by picking up the decision-maker’s intuitive view in terms of ordinal information of evaluation criteria and alternatives. However, the fact that this method is partitioning the total solution space in order to identify feasible solutions was not thought by the experts as to encourage transparency. This can also be confirmed from the application example where the loss of information derived from the transformation of cardinal to ordinal data has led to a final ranking (A1>A2>A3) being different from the prevailing (A3>A2>A1) identified by the majority of the other methods. The ELECTRE family have certain limitations that, to an extend, affect transparency. These are strongly related to the existence of thresholds which are arbitrary defined. The dependency of these thresholds to the final rankings can also be seen from the application example, where a small modification of the discordance limit ε* (0.3 instead of 0.1) has resulted to a rank reversal (A3>A2>A1 instead of A2>A3>A1). Finally, the additive model approaches appear as the most favourable. From an applicational point of view, the methods are considered in position to cope with almost any problem in transport appraisal. The concept of trade-offs that these models are implementing was well accepted by the experts. AHP and MAUT proved to be mathematically more straightforward than the ADAM method, which is more complicated. The rankings obtained by the additive models were quite reasonable with the exception of AHP that produced an output (A3>A1>A2) which can be considered to be a compromise solution.
13
Simplicity One of the most desirable characteristics of any assessment methodology is its simplicity. A method’s ability to provide a crisp and well-defined methodological representation of complex real world decision situations could be seen as a possible measure of implementation simplicity. The additive methods proved relatively simple during the final phase of the assessment (although, in the application example, no particular difficulty was encounter even during the determination of the utility and value functions). Moreover, the AHP method is simple throughout, due to the fact that no utility functions are used. The ELECTRE methods are slightly more complicated and require more attention in complex situations (e.g. when non-transitive superiority relations are used). However, no such cases were encountered in the application example. The vectorial methods are not particularly difficult to implement, yet certain side effects may cause difficulties in supervising REGIME. This is usually due to the complicated nature of a vectorial space, especially when a uniform representation of it, is attempted.
Robustness Robustness is a very important property especially when it comes to strategic evaluation of transport networks, where most decisions are taken in uncertainty using diverging information. As stated above, robustness refers to the operational efficiency of a method and as such is touching upon the following methodological points.
Data Requirements The REGIME method is able to analyse ordinal and cardinal data and therefore within a multi-objective framework, it can manage a large variety of assessment problems. However, as shown from the application example this might lead to a considerable loss of information when the available data has be transformed from a cardinal to an ordinal scale. The superiority graph models are mainly quantitative approaches dealing with qualitative information by first transforming it into cardinal data. No ordinal information or verbal statements can be used directly. Since the impact matrix of the application example was purely quantitative no such problems were encountered during the implementation of the ELECTRE methods.
14
The methods using an additive function are also quantitative in terms that no evaluation can occur if all the available information is not first transformed into cardinal. Using the utility theory typology to qualitative multicriteria evaluation, this approach is called the indirect approach. TABLE 4
Treatment of any Number of Projects/Criteria The additive methods, due to the underlying assumptions of utility measurements and the properties of the ideal-point approach can consider any number of projects in an independent fashion. The ELECTRE methods, may cause decision diffusion when the number of competing projects is increasing. This diffusion that exists due to the variable degree of permitted compensation can be avoided if the Concordance and Discordance limits are shifted to more "flexible" thresholds. However, this has negative consequences on simplicity since it requires continuous resetting of these limits. The REGIME method can theoretically treat any number of projects/criteria. However, as the number of projects increases, the complexity in the computation of the final evaluation charts also increases disproportionally. With respect to the number of criteria, the REGIME analysis is also cumbersome when this number increases. This is due to the fact that in a vectorial space of a high order it becomes computationally very difficult to identify the sub-spaces with the same sign of differences between alternatives. In contrast, the ELECTRE methods perform very well to a large number of criteria, provided that the associated quantitative weights have been determined from the beginning. Finally, the additive methods perform well to any number of evaluation criteria. For the methods using utility functions, the larger the number of criteria, the larger is the number of such functions. The same also holds for the ideal-point approaches (i.e., the larger the number of criteria the larger the number of point co-ordinates that have to be computed). An exception here is AHP, which in theory can treat any number of criteria, but in practice it is very difficult for the decision maker to pairwise compare numerous criteria in a consistent way. Potential problems of intransitivity of pairwise comparisons between numerous criteria can be overcome by making use of the “Consistency Index” (proposed by Saaty, 1980).
15
One concern about using pairwise comparisons for weights’ estimation is that the relative weights achieved are strongly influenced by the particular set of criteria considered, (e.g., if other evaluation criteria are added at a later stage, there could be preference reversals, particularly if they have more extreme scores).
Treatment of Uncertainty The REGIME method starts with an arbitrary qualitative ordering of evaluation criteria. By doing so the method picks up the decision-maker’s intuitive view of the relative risks involved in weighting a set of criteria and ranking a set of alternatives, through the holistic comparisons it requires. The same also holds for the ELECTRE methods. Thus, the determination of criteria weights takes place arbitrarily, bearing in mind the same assumptions. As regards the additive models, MAUT treats uncertainty through appropriate utility transformations of impact scores in the Keeney/Raiffa tradition (Raiffa, 1970). This can be seen from the application example, where the partial utility functions have been defined on the basis of properly selected characteristic points. AHP and ADAM methods are using the arbitrary qualitative ordering of evaluation criteria as in the vectorial and outranking methods.
Encouragement of Special Interest Groups The REGIME method does not use any group-based decision making techniques. In fact, it can not combine different weight profiles in one single run. Instead, the method has to be repeated several times (one for each actor) to provide rankings that could be proportionally taken into account by the decision maker in the process of reaching to a final decision. ELECTRE I, II and III also share the same considerations like the REGIME and as such are limited to convey one single decision maker’s view. ELECTRE IV, however, avoids using criteria weight on the assumption that a single-point solution (e.g. weights) tends to lead to deadlocks in a decision process by imposing too rigid conditions to reach a compromise. A weak point of this approach is that it might require an excessive number of globally accepted evaluation criteria, the identification of which may not be an easy task.
16
Among additive methods, AHP is the most advanced since it aggregates the preference rankings of individuals into a consensus ranking. Group-AHP uses an additional level in its hierarchical structure consisting of stakeholders. Each stakeholder states his/her preferences which are combined together according to the degree of importance (power or knowledge) of the individual making the judgement. Thus, a final ranking is computed.
Sensitivity Sensitivity here refers to the ability of each method to pick up differences in projects’ performances when the differences between criteria weights and project attributes, are very small. This problem is two-fold and it is related: (i) to the quality of the transformations from a physical to an artificial scale of measurement, and (ii) to the minimum “resolution” of each method considered. As it has proved from the application example, the additive methods show the best characteristics in terms of sensitivity. The ability to use utility and/or value functions, which might be linear or non-linear, direct or indirect, offers a series of advantages to the additive models that cannot be found in any other method. On the other hand, the superiority graph models use a completely different approach. The ELECTRE methods for example, can be linear and very descriptive as regards to the criteria’ relevant weights, whereas for the project performance they can be as sensitive as the so-called “veto threshold” allows. The “veto”, which here defines the minimum acceptable resolution between two projects in order to consider their outranking relation as valid, is acting as a “security valve”. The REGIME method is using ordinal information to structure an evaluation problem. This makes almost impossible to take into account the intensity of preferences and thus, assume smooth transitions between criteria weights and project performances. An attempt to overcome this problem was made by Hinloopen (1983) who suggested the use of intermediate values (e.g. +1/2 or -1/2) in order to define the regime components in a more flexible way.
Accountability An accountable multicriteria analysis, besides its overall linkage to the ability of tracing back a decision, on a methodological level it is related to the degree of compensation between alternatives. This can be better understood
17
through the concept of incomparability. In decision analysis incomparability relations usually render the assessment process more realistic and enhance the accountability of results. Nevertheless, these relations can only be modelled when no complete compensation is allowed (i.e. when the criteria attributes do not counter-balance each other when combined together). Thus, one may assume that, in the majority of cases, accountability and compensation are reverse concepts. In this context, the superiority graph models can be seen as the most accountable methods since they can control the degree of compensation by properly fixing the veto thresholds. The additive models, on the other hand, allow for complete compensation. The diffusion that in some cases is caused by trade-offs may obstruct the decision maker to trace back a decision. The same considerations also apply for the vectorial methods. REGIME is aggregating on the basis of a linear additive model using only ordinal information. This makes decision trace-back even more difficult than in the additive models. A summary of all findings reached through the comparative analysis of the methods is given in table 5 below. TABLE 5
CONCLUSIONS A comparative analysis of five well-established multicriteria methods has been presented in this paper in order to help the potential decision maker assess their ad hoc suitability for the appraisal of transport initiatives. The main conclusions drown from this study can be summarised as follows: 1.
The theoretical background of each method is philosophically consistent with the decision-making framework of transport planning.
2.
The methods are capable of receiving inputs concerning preferences of the actors involved and they can generate outputs permitting the evaluation of direct impacts as well the assessment of indirect effects on social and physical environment.
3.
The methods are relatively easy to use by the decision-makers and have the potential to be a decision support tool for the selection among different transport initiatives.
18
A panel of experts was employed to examine the merits and shortcomings of the competing multicriteria methods on the basis of an application example consisting three infrastructure projects in Greece. This was done in order to show their usefulness as well as their limitations in terms of transparency, simplicity, robustness and accountability. From the comparative analysis it has been shown that:
each method presents a series of unique features allowing for a high degree of flexibility, consistency and reliability,
the methods' performance depends on the characteristics of the decision situation In summary, the main positive and negative aspects of each method are the following:
REGIME: useful when ordinal information is available for criteria weights and projects’ scores. The method is a powerful tool with the ability to manage a large variety of evaluation problems. The number of criteria plays an important role in determining the implementation difficulty.
ELECTRE family: They are based on the partial comparability axiom. ELECTRE I may lead to inconsistent results when “non-transitive” outranking relations are used. Most of these problems have been solved in latter versions.
Additive Methods: the most straightforward, close to human rational, methods to treat transportation decision problems. There is a variety of decision tools using utility or value functions (MAUT and AHP) or the notion of the ideal point (ADAM type). The additive models are usually linear and allow for complete compensation. However, this is not desirable in all decision situations. Summarising, this paper has presented a comprehensive analysis of the properties of five commonly applied MCA methods. The intention was not to seek for a globally “optimum” method for transport assessment since no such method exists, but to provide to the decision-maker the information he/she might need in order to select (when necessary) a method that would adhere to the specific characteristics of the assessment problem he/she might need to solve.
19
REFERENCES: Bana e Costa, C. A. (1990). “Readings in multiple criteria decision aid.” Springer-Verlag, Berlin. Banister, D. (1997). “Decision Makers Requirements for Assessment.” European Commission, SAMI Project, Working Paper 1. Beuthe, M. Scanella, G. (1996). “Applications comparees des methodes d analyse multicritere UTA.” R.A.I.R.O. Operations Research, vol. 30, 3, pp.293-315. Bouyssou, D. Perny, P. (1990). “Ranking Methods for Valued Preference Relations: a Characterisation of a Method based on Leaving and Entering Flows”. Cahier du LAMSADE, n.101, Paris. Hinloopen, E. Nijkamp, P. Rietveld, P. (1983). “Qualitative Discrete Multiple Criteria Choice Models in Regional Planning.” Regional Science and Urban Economics 13, North Holland, Publishing, pp 77-102. Mintzberg, H. (1979). “The structuring of organisations: a synthesis of the research.” Prentice Hall, New York. Munda, G. (1995). “Multicriteria Evaluation in a Fuzzy Environment.” Physica-Verlag, Heidelberg, Germany. Nijkamp, P. Rietveld, P. Voogd, H. (1990). “Multicriteria Evaluation in Physical Planning.” Contributions to Economic Analysis, North-Holland, Amsterdam. Nijkamp, P. Blaas E. (1993). “Impact Assessment and Evaluation in Transportation Planning.” Kluwier Academic Publishers, Dordrecht. Pearce, D. W. (1978). “The valuation of social cost.” Allen and Unwin, London. Raiffa, (1970). “Decision Analysis Introductory Lectures on Choices Under Uncertainty.” Addison-Wesley, USA. Roy, B. (1985). “Methodologie Multicritere d Aide a la Decision.” Economica, Paris. Saaty, T. L. (1980). “The Analytical Hierarchical Process.” New York, Wiley. Schaerlig, A. (1985). “Decider sur plusieurs criteres.” Presses Polytechniques Romandes, Lausane. Szidarovsky, M. E. Gershon, L. Duckstein (1986). “Techniques for Multiobjective Decision Making in Systems Management.” Elsevier Science Publishers, Netherlands. Tsamboulas et al (1998). “Assessing the Socio-Economic and Spatial Impacts of Transport Initiatives: The EUNET Project.” Proceedings of the 8th World Conference on Transport Research, Antwerp, July 12 to 17, 1998, Elsevier Science Ltd., Netherlands. Willis, K. G. (1980). “The Economics of Town and Country Planning.” Granada, London.
20
Zeleny, M. (1982). “Multiple Criteria Decision Making.” McGraw-Hill Book Company.
21
Table 1: Relative classification of the candidate methods Decision-theory typology Mathematical Structure
Weighted linear
Outranking
Multi-Attribute Utility
additive models
methods
Theory (MAUT)
the vectorial
REGIME
Ideal-point approach
model ELECTRE Group
the superiority graph model the additive
AHP
Multiattribute utility approach
model
22
ADAM type
Table 2: The matrix of alternatives and criteria Criteria
Units
IRR
% % 1-100
Safety Environment
Alternatives
Weights
Mitilini Port
Patras Port
Rion-Antirion bridge
12 5 4
10 8 9
7 9 10
23
0.40 0.35 0.25
Table 3. Summary of rankings according to the applied MCA methods METHOD IMPLEMENTED
PROJECT’S RANKINGS
The vectorial model
REGIME
A1>A2>A3
ELECTRE I
{A2, A3}>A1 A2>A3>A1
The superiority graph model
ELECTRE II
ELECTRE IV
A3>A2>A1 A3>A2>A1
ADAM
A3>A2>A1
MAUT
A3>A2>A1
AHP
A3>A1>A2
ELECTRE III
The additive model
*
from best (left) to worst (right)
24
Table 4: Data requirements of the multicriteria methods REGIME
ELECTRE
Qualitative data Ordinal
MAUT
AHP
ADAM
data Cardinal
data
25
Table 5: Summary of performances of the five multicriteria methods Criteria
REGIME
ELECTRE
MAUT
AHP
ADAM
Q1
*** * * * ** ** * * ** * *
** ** ** ** * ** *** * * ** ***
** *** ** *** ** *** *** *** * *** **
*** *** *** *** *** *** *** ** *** *** **
** *** ** * ** *** *** ** * *** **
Q2 Q3 Q4 (S) (P) (C) (U) (D) (V) (A)
*** very good, ** good, * not good. (Q1) copes better with real world situations (Q2) offers decision analysis the closest to human rational approach (Q3) is well structured and easy to follow (Q4) uses straightforward mathematical approximations to represent reality (S) Simplicity (P) Treatment of any number of projects (C) Treatment of any number of criteria (U) Treatment of uncertainty (D) Encouragement of special interests group (V) Sensitivity (A) Accountability
26
B
C G E F
D
A
Figure 1: Representative alternatives in the case of two criteria. Note: Point C is the ideal point. The non-ideal point is D, whereas G, E, F represent efficient alternatives located on the far end of the efficient set. Alternative E can be considered as the best compromise solution.
28
Key Words: Multicriteria Analysis, Project Assessment, Multicriteria Methods, Transport Appraisal, Comparative Analysis.
29
