THE PROS AND CONS OF USING PROS AND CONS FOR MULTI-CRITERIA EVALUATION AND DECISION MAKING 26 October, 2009
Michael Wood Department of Strategy and Business Systems University of Portsmouth Business School Richmond Building Portland Street Portsmouth, PO1 3DE, UK.
Tel: +44-2392844168 Fax: +44-2392844037
[email protected] http://userweb.port.ac.uk/~woodm/
1
THE PROS AND CONS OF USING PROS AND CONS FOR MULTI-CRITERIA EVALUATION AND DECISION MAKING Abstract Many methods have been proposed for helping with multi-criteria evaluations and decisions. This paper discusses one very simple method – the idea of evaluating objects by comparing them with a base object, and assessing their Pros And Cons On (one or more) Common Scales (of value), summarised by the acronym PACOCS. If the objects in question are the alternatives in a decision, PACOCS can be used to assist decision makers choose the best option. The paper argues that, in many contexts, PACOCS is closer to common sense and its rationale more transparent and user-friendly than other approaches to multi-criteria problems based on weighted sum functions and “even swaps”, despite broad similarities in mathematical terms. This additional transparency is of particular potential importance in the many situations where people untrained in decision analysis have to make multi-criteria decisions. Key words: Multiple criteria decision analysis, Decision analysis; SMARTS, Even swaps, Value function.
2
Introduction Everyone faces a continuing stream of multiple criteria decision problems in their personal and business lives, so ideas about how these problems should be approached are relevant to a very wide range of people in very wide range of contexts. However, the methods that have been proposed, and the rationale behind them, are, in many cases, relatively complex. This means that there is a danger that methods may be used without sufficient attention to the precise inputs required, and the assumptions on which sensible use of the methods depend. Many approaches to analyzing multiple criteria decision problems involve expressing an estimate of the overall value of the possible options as a function of a number of "partial value functions" representing the criteria to be taken into account in the decision (Belton and Stewart, 2002). In the simplest version of this model, this function is linear with constants typically known as "weights", and the resulting overall values are, in effect, weighted averages. A major difficulty here is the definition, assessment and interpretation of the weights (Belton and Stewart, 2002). The purpose of this article is to propose a simple approach to measuring overall value and assessing which option is best – called PACOCS for reasons to be explained below. Most new methods proposed in the literature are more powerful and more complex than those they seek to replace or supplement. My argument in this paper is that PACOCS is very similar to established methods in terms of its power to solve problems – the close mathematical similarities are explained below – but that it has the potential advantage of being simpler in the sense that, in many contexts, it is more transparent and closer to common sense. This is likely to encourage wider use, discourage silly errors (the rationale is so clear that these should be obvious), and to facilitate the closer integration of formal methods and human reasoning which is the key feature of the “new paradigm” extolled by Beynon et al (2002). As the number and complexity of avalailable techniques increases, it becomes more and more necessary to take any opportunity to simplify techniques (Wood, 2002). Hajkowicz and Higgins (2008) advocated the use of the weighted average method (or “weighted summation”) in preference to more complex methods because of its simplicity. My argument here is that PACOCS is even simpler, and so more useful. There are many situations in everyday and business life where overall value is assessed as a function of performance on individual criteria – e.g. quality assessments for small 3
businesses (Xu and Yang, 2003), evaluations of universities in ranking lists such as the Times Good University Guide (Times Online, 2007), effectiveness of countries in tackling environmental problems, a ranking of “the best companies to work for” (Sunday Times, 2006), defining an overall grade for students on a course (Bouyssou et al, 2000) and choice of a “best buy” consumer product (e.g. http://www.which.co.uk). In situations like these, with no expert decision analysis advisor available, any additional transparency in the methods used is likely to be especially valuable. The next section outlines an example which will be used to illustrate how PACOCS works, and then we will discuss some of the alternative methods and how PACOCS compares with them. (There is a summary of
PACOCS for a non-specialist audience
at
http://userweb.port.ac.uk/~woodm/McdaPacocs.pdf .)
An Example I will illustrate the argument by an analysis of the decision about which mode of transport to use for my ten mile journey to work. One of the main benefits of decision analysis is that it helps interested parties understand the factors which lead the rational decision maker to make one choice rather than another: with the present transport situation there are obvious benefits in understanding these factors, both in my own case, and in the case of the other people whose cars clog up the town where I work. This is, of course, just one example: the argument of this paper applies to the full range of applications of multiple criteria value functions. I will start from Table 1. The cost figures in this table include all relevant costs – including the costs of the necessary refreshments if I were to cycle, and the capital cost of having a car available to drive. Some entries in this table may strike the reader as idiosyncratic – but they are, of course, my personal assessments. Table 1 assumes that we have clarified the context of the decision, the options available and the criteria by which these will be compared, checked that the criteria are structured in an appropriate manner (e.g. that they are comprehensive and don’t overlap), and eliminated any dominated alternatives – this final step led to the elimination of the option of walking to work which was worse than cycling on several criteria and better on none.
4
Table 1: Objects by attributes matrix (or consequences table) Objects of evaluation Attribute
Go by car
Take train
Ride bike Take taxis
Cost (per day)
£14.38
£2.50
£2.52
£20.00
Comfort
Good
Good
Bad
Good
Ability to carry luggage
Good
Bad
Moderate Good
Ability to work on journey
Bad
Good
Bad
Bad
Duration (minutes)
20
50
50
30
Flexibility
Good
Timetable? Weather? Available?
Risk of delays
Good
Bad
Puncture? Good
Fitness
Bad
Moderate
Good
Bad
Accidents
Moderate
Good
Bad
Moderate
Pollution
Bad
Good
Bad
Bad
Bad
Good
Good
Moderate
Quality of journey
Timing issues
Health costs / benefits
Environmental impact
In many cases presenting the information in this form may be sufficient, but sometimes a formal way of dealing with this may be helpful. The next section explains the proposed approach.
PACOCS The approach proposed in this article is to assess the Pros And Cons On one or more Common Scales of value (PACOCS). The procedure is to choose an object as the base, and then measure differences in value on each criterion between this base and the other objects using one or more common scales of value. These differences – pros if positive and cons if negative – can be added up to find the aggregate difference for each value scale. If there is just one value scale this aggregate can then be used as a basis for comparing the overall value of each object. The idea behind the approach is very obvious, but deserves to be taken seriously. In the case of a decision problem, the objects in question will be the options open to the decision maker; in the case of an evaluation problem the objects will be the entities being compared. The base may be one of the actual objects being compared, or it could be a 5
hypothetical object introduced for the purpose. With the transport decision, taking the Ride bike option as the base, and using money as the (single) common scale of value, the data in Table 1 leads to the results in Table 2. The train option, for example, is valued at £7.02 more than the bike option. This is simply the sum of £0.02 for the cost attribute (the train is cheaper so its cost is better by this margin), £2.00 for the comfort attribute (corresponding to extra value I associate with the better comfort of car, train and taxis compared with bike), and so on for all the attributes. The importance of each attribute is implicit in these values – the difference between bike and train on the Ability to work attribute is valued at £5, compared with only £2 on the comfort attribute. The figure for the luggage attribute is negative, corresponding to the fact that I considered the train less convenient for carrying luggage (bikes are surprisingly good in this respect). Table 2 makes it clear how the other objects compare with riding the bike: e.g. the biggest pros for the train, compared with the bike, are the ability to work on the journey, and the reduced risk of accidents.
Table 2: Value of each option relative to Ride bike Value of each option Attribute
Go by car
Take train
Ride bike
Take taxis
-£11.86
£0.02
£0.00
-£17.48
Comfort
£2.00
£2.00
£0.00
£2.00
Ability to carry luggage
£1.00
-£1.00
£0.00
£1.00
Ability to work on journey
£0.00
£5.00
£0.00
£0.00
Duration (minutes)
£4.50
£0.00
£0.00
£3.00
Flexibility
£5.00
£0.00
£0.00
£2.00
Risk of delays
£2.00
-£5.00
£0.00
£3.00
-£7.00
-£2.00
£0.00
-£7.00
£5.00
£8.00
£0.00
£5.00
Environmental impact
-£6.00
£0.00
£0.00
-£3.00
Aggregate advantage over Ride bike
-£5.36
£7.02
£0.00
-£11.48
Cost (per day) Quality of journey
Timing issues
Health benefits Fitness Accidents
This procedure has much in common with the additive difference model (Tversky, 1969), with methods involving additive value functions, and with a number of other approaches, 6
some of which are discussed below.
Eliciting the pros and cons Any method of multi-criteria decision analysis requires the elicitation of judgments from the decision maker: for example, Table 2 includes my judgment that going by car is worth £4.50 more than cycling from the perspective of the duration of the journey. The saving in time is 30 minutes (see Table 1) which I have judged is worth £4.50 to me. Subjectively, these judgements are very difficult to make: they feel arbitrary and unreliable. Clearly an analyst eliciting judgements such as these from a decision maker would need to phrase the questions carefully, take account of the fact that the answers may not be stable, and perhaps run a sensitivity analysis to check the robustness of the model. If we are prepared to assume a proportional relationship between the value of time saved and the duration of the journey, then it is a trivial matter to determine a conversion factor for converting time saved to its value in pounds – in the case of Table 2 this factor is £0.15 per minute, which allows us to conclude that the 20 minutes saved by taking a taxi rather than a bike is worth 20 x £0.15 or £3.00. Similarly if we were prepared to accept the additional carbon emissions for each journey as an adequate indicator of environmental impact, then this scale could be derived in a similar way. Regardless of whether these judgments are made directly, or by means of a conversion factor, we now need to ask how we can decide if these judgments are right. This is, of course, a difficult question – but a question of a type that any multi-criteria method must face. On one interpretation the question presupposes a “truth” against which the judgment can be judged; the difficulty being, of course, the lack of any obvious alternative method of finding this truth to serve as a check – the best that can normally be done is to check consistency by asking how stable the judgment is or how “comfortable” the decision maker feels with it. At the other extreme, we might take the judgment as a definition, or construction, of value – and the resulting model describes how I should behave on the basis of assumptions like 30 minutes saved on the journey being worth £4.50. The main virtue of the PACOCS formulation of these judgments is that they are so clear that difficulties of interpretation are likely to be obvious. We have assumed so far that the pros and cons are derived from the criteria and other information in Table 1. However, this sequence of events could be reversed, as considering
7
the pros and cons of different objects may be a good way of eliciting the criteria on which Table 1 is based. One well known package (Expert Choice) for assisting multi-criteria choice, for example, incorporates a “ProCon pane” for users to type in pros and cons which are then converted (manually) into attributes (French and Xu, 2005). The idea behind PACOCS is to stay close to this initial structuring process – a vital part of any decision making procedure (Journal of the Operational Research Society, 2006). We turn next to two issues which have an obvious bearing on the judgments – the choice of the scale and the base.
The choice of common scale(s) of value The scale(s) used in a PACOCS analysis needs to be a ratio scale with a meaningful zero corresponding to the situation where there is no difference between the given object and the base. In practice, this would normally be a numerical scale where positive numbers represent pros, and negative numbers represent cons. The chosen (numerical) scale in Table 2 is money, which is often the most convenient scale to choose. There are, however, a number of problems with using money in this way. The relationship between money and value may not be a linear one (e.g. the economist’s diminishing marginal utility may apply); there may be differences between the market price and the price an individual is prepared to pay which makes money ambiguous as a measure of value; and there are arguments that decision makers find it difficult to compare financial criteria with nonfinancial ones – according to Edwards and Newman (2000: 24), “early translation of nonmonetary effects into money terms tends to lead to under-assessment of the importance of nonfinancial consequences.” Any of the other criteria could be used as a common scale for Table 2. Taking Duration as an example, we could define one “10 minutes earlier” (TME) point as being the extra value of getting to work in 40 minutes rather than the 50 minutes of the journey by bike. If we assume a linear relationship between duration and value over the range from 20 minutes to 50 minutes, the value of the shorter duration of the taxi trip (30 minutes) is 2 points, and of the trip by car (20 minutes) is 3 points. Then, the decision maker could be asked to use these points to evaluate the differences between the bike journey and the other modes of transport on each of the other criteria – e.g. the comfort differential between Ride bike and the other modes of transport might
8
come out as 1.3 TME points. In principle, a similar procedure could be used to measure value differences by a measure based on any of the other criteria – e.g. a comfort point would be the difference in comfort between the bike and the other modes of transport. However, using some of these criteria as a basis for a scale of value would be decidedly awkward, in terms of both elicitation (the questions posed are not particularly easy to make sense of) and interpretation. It is difficult to see a way of avoiding these difficulties. Other methods of analysing multi-criteria decisions face the same problems and resolve them in fairly similar ways – e.g. the swing weighting procedure (discussed below) involves similar concepts. The obvious general advice is to try and ensure that the scales used are the most appropriate taking into account the difficulties of elicitation and interpretation. It might also be helpful to take account of the inevitable uncertainty in these measurements by means of fuzzy logic (which has been used in a variety of multiple criteria applications – e.g. Perris and Labib, 2004), or by using an interval rather than a single value – e.g. we might describe the comfort differential between Ride bike and the other modes of transport as between £0.10 and £5.00 instead of £2.00 on the cost scale.
PACOCS with more than one scale of value Sometimes it may be natural or useful to combine the pros and cons from some attributes but not others. Table 3 shows the pros and cons aggregated for each of three scales representing Cost, Personal journey benefits and Environmental impact. The first is measured in money, for the second I have used TME points (see above), and for the third “environmental impact points” (The values for the first scale are identical to Table 2, for the others, values are in the same proportions as Table 2 because I have derived them from Table 2.)
Table 3: Values relative to Ride bike in different scales Value of each object Attribute
Go by Take train Ride bike
Take
car Cost: £s per day)
taxis
-11.86
0.02
0.00
-17.48
Personal journey benefits (Quality, Timing, Health): TME points
8
5
0
6
Environmental impact (Environmental impact points)
-6
0
0
-3
This could either be the end of the formal analysis—it is a simpler model than Table 1 and so may be more helpful to the decision maker in coming to a decision – or it might 9
represent an intermediate level of analysis which could be progressed in different ways – perhaps experimenting with different trade-offs in different contexts.
The choice of base object The choice of base is another issue of obvious importance. The reason for choosing the bike as the base in Tables 2 and 3 was that this was the main mode of transport I used before the analysis, so it was the natural base to choose to see if it would be worthwhile switching to one of the other modes of transport. A second example, discussed briefly below, involves a comparison of different universities; a convenient base to use here is a hypothetical university with the median rating on each criterion. And the base implicit in our comparison of PACOCS with alternative methods of analysing multi-criteria decision is PACOCS itself (below). Clearly the natural base will depend on the context and purpose of the analysis. On the assumption that decision analysis reflects, rather than creates, an underlying truth about the decision maker’s preferences, we would hope that the order in which the objects are ranked on any scale would not depend on the base object chosen. Under some circumstances this may indeed be the case – e.g. if all value differences are derived from numerical attributes with conversion factors which are the same regardless of the choice of base object, then the resulting model will be a linear one and the results will be identical, apart from the zero on the scale, for any choice of base. But if parts of the model are not linear, or if the conversion factors are different for different bases, then the order in which objects are ranked may be different for different choices of base. (This follows from the intransitivity theorem in Tversky (1969). This shows that the additive difference model leads to the possibility of intransitivity unless all the scales are linear; this cannot be true of PACOCS with a single scale of value because each object has an aggregate advantage over the base on the same numerical scale, but it does show that there may be Objects A, B and C such that A is preferred to B, and B to C, but C to A. If C is the base object for PACOCS, then B will have a positive advantage, C will score 0, and A will score negatively. If, on the other hand B were the base object, then A will score positively, B will score 0, and C will score negatively. In other words, the order in which objects are evaluated may depend on the choice of base object.) There is a considerable amount of empirical evidence for one type of non-linearity – the fact that gains may be evaluated differently from losses (see, for example, Bell et al, 1988: 24-5; 10
Gilbert, 2007: 145-6). This might mean, for example, that having a base at the bottom of the scale so that all comparisons are gains may give different results from having it in the middle so that some are gains and some are losses. These points suggest that the choice of base may make a difference to the results, so there would be an argument in favour of checking by running the analysis with a different base.
Assumptions underlying PACOCS Using PACOCS to measure the relative value of a group of objects obviously entails making some implicit assumptions. First, and most obviously, the model is an additive one in the sense that the pros and cons are added and the resulting sum is taken to be a sensible measure of the combined effect. Adding pros and cons obviously provides a means by which a disadvantage on one criterion can be compensated for by an advantage on another criterion: in other words tradeoffs can be made between the criteria. There are obviously other ways in which pros and cons could be combined besides summing them (e.g. we could use a product): additive models have the advantage of simplicity, and experience shows that they are useful across a wide range of contexts (Belton and Stewart, 2002: 86, 103). Davis and Hersh (1981), in a discussion of additive models across a wider domain, stress the importance of convenience, and conclude that “any systematic application of addition to a wide class of problems is done by fiat” (p. 74)— “we deliberately force various physical and social aspects of the universe …” (p. 70) into convenient mathematical models. The second important, implicit assumption is that judgments of pros and cons can be made independently of each other. In our example the conversion factor for Duration is –£0.15 per minute, indicating that every extra minute of journey time decreases the value of the journey by about 12 pence. However, while this may be reasonable for journeys by car or bike, when going by train the value of extra time could conceivably be positive if I value the opportunity to work in peace on the train. If this were the case, I would need to make a separate assessment of the advantages of shorter journeys for each different mode of transport: this would obviously make the model more complicated. Similarly, the use of more than one scale gives the opportunity to avoid compromises between different sets of criteria, so the extent to which PACOCS is compensatory is flexible; and the pros and cons could, in principle, be combined by a method other than addition. The PACOCS method can obviously be adjusted in a number of different
11
ways. Both of these assumptions should be clear from the process of setting up Table 2. If the values in the cells for the duration attribute were not appropriate, or if the sum of the columns did not represent a sensible aggregate, then the headings in the table should alert the decision maker to the difficulty.
A second example – The Times University Subject Tables The UK newspaper, The Times, publishes a list of ratings of British universities in different subjects which are designed to help prospective students choose a suitable university (Times Online, 2007). For example, the Business subject rating gives Oxford University a score of 100 and Portsmouth University a score of 81.5. These scores are based on three criteria: “research Quality, Entry Standards and Graduate Prospects. … These were combined using a z-score transformation with equal weighting for the indicators and the totals were transformed to a scale with 100 for the top score.” Using a PACOCS analysis with the same three criteria, with the base being the “median university” – a hypothetical university which has the median score on all three criteria – and the common scale being a “points” scale where a point corresponds to a 10% improvement in the graduate prospects, the score for Oxford is 8.5 points above the median university, and for Portsmouth 1.7 points above the median. These scores are derived by summing scores for each of the three criteria: the score for Oxford comes from 2.1 for the first criterion, 1.9 for the second and 4.5 for the third. (This analysis assumed linear conversion factors, with a potential decision maker being consulted about their value.) The advantage of the PACOCS formulation here is that it seems more likely to prompt users to question the rationale behind the scores – why these three criteria and why is it sensible to add scores for each? This might then lead to the formulation of a model more in tune with the real values of a potential decision maker.
Other approaches PACOCS incorporates aspects of a number of well established approaches to analysing decisions. One of the earliest to be explicitly described is in a letter from Benjamin Franklin, dated 1772, on his "moral or prudential algebra" (Hammond et al, 1998). This involves listing 12
the pros and cons of a proposal on opposite sides of a sheet of paper, and then trying to "strike out" single entries or groups of entries on each side that he considered of equal force. In this way he could come to a rough decision about how the two sides balance. In essence, this idea of listing and comparing pros and cons of the available options is a very common and natural approach to analyzing decisions. As well as PACOCS, it is the underling idea of SWOT (strengths, weaknesses, opportunities, threats) analysis. PACOCS as implemented in Table 2 (but not Table 3) also has much in common with the idea of "pricing out" each criterion in terms of money. Clemen (1996) considers this, but points out that it is sometimes difficult or may seem inappropriate: pricing out costs and benefits of medical procedures may seem "cold-hearted" (p. 21), in which case, he claims, other procedures may be easier (p. 547). Pricing out is similar to cost-benefit analysis (CBA) which is an approach developed by economists to measure costs and benefits of a proposal on a monetary scale. It is then easy to add these up to ascertain whether there is a net benefit or a net loss, so, in principle, the multi-criteria problem is solved. There are many other approaches to multiple criteria decision analysis which take a different route from PACOCS. Some of these are designed for particular situations (e.g. the method used for the Times University Subject Ratings described above), whereas others are intended to be applied across a range of situations. We will now consider a few of these general approaches, focusing on those which are sufficiently similar to PACOCS to make a comparison interesting.
Comparison with “SMARTS” and other MAVF approaches One commonly advocated approach to analyzing situations like the one depicted in Table 1 is to devise a value scale for each attribute going from 0 representing the worst value to 10 or 100 representing the best, and then to use a weighted sum of these partial values for the aggregate function. The weights are derived from the “swing weighting” procedure which involves asking the decision maker to compare the values of a swing from worst to best on each of the criteria. This approach has been dubbed SMARTS – "Simple Multi-attribute Rating Technique ... using Swings" – by Edwards and Baron (1994). Table 4 summarises the results of this applied to the situation in Table 1.
13
Table 4: Partial values, weights and aggregate values (SMARTS) Single-dimensional value
Normalized Attribute
weight
Go by car
Take train
Ride bike
Take taxis
25%
3.2
10.0
10.0
0.0
Comfort
1%
10.0
10.0
0.0
10.0
Ability to carry luggage
3%
10.0
0.0
4.0
10.0
15%
0.0
10.0
0.0
0.0
Duration (minutes)
5%
10.0
0.0
0.0
7.0
Flexibility
8%
10.0
0.0
0.0
5.0
10%
8.0
0.0
5.0
10.0
6%
0.0
5.0
10.0
0.0
Accidents
15%
5.0
10.0
0.0
5.0
Pollution
5%
0.0
10.0
0.0
0.0
6%
0.0
10.0
10.0
5.0
4.0
7.2
4.4
3.2
Cost (per day) Quality of journey
Ability to work on journey Timing issues
Risk of delays Health costs / benefits Fitness
Environmental impact Aggregate values
There are many variations on this theme: for example, the swings used for defining the weights might start in the middle of the scale rather than the bottom, or the status quo might be used as starting point for the swings (Belton and Stewart, 2002), or different terminology might be used—e.g. Keeney (1992) avoids the word "weight" referring instead to "scaling constants that indicate the value trade-offs" (p. 133). Another possibility is to ask the decision maker to assess trade-offs directly (see, for example, Fischer, 1995; Poyhonen and Hamalainen, 2001) and then use these to infer the weights. We will take SMARTS and Table 4 as a representative of this general approach. In many ways, SMARTS is similar to a PACOCS analysis taking a hypothetical object which is worst on all criteria as the base. The SMARTS aggregate value for the jth object, SVj, is given by SVj = v1 j w1 + … + vn j wn
Equation 1
where the normalised weights are w1 , … wn and the partial values, standardised on a 0 to 10 scale as in Table 4, are v i j. For those criteria where these partial values depend on a numerical 14
attribute, and we are prepared to assume a linear relation between this attribute and its partial value, then we can also write vi j = ki ∆xi j
Equation 2
where ∆xi j is the difference between the value of the attribute for the jth object and the worst value of the attribute (i.e. the value corresponding to the PACOCS base object), and each ki is a constant depending on the size of the swings on the attribute and value scales. The PACOCS value (assuming a single scale of value), relative to the base, for the the jth object, PVj, is given by PVj = ∆v1 j + … + ∆vn j
Equation 3
where ∆vi j represents the value of the jth object on the ith criterion relative to the base. As for SMARTS, where we can assume a proportional relation between an attribute and value, we also have ∆vi j = ci ∆xi j
Equation 4
where ci is the conversion factor for the ith attribute. Equations 1-4 and their derivation show the similarities and differences between SMARTS and PACOCS. Both are additive models, and with respect to attributes measured on a numerical scale, they are linear models if we are prepared to make the necessary assumptions about a linear relationship between the attribute and value. Furthermore, both models will obviously give consistent results if the judgments made in building each are consistent. In fact the judgments made in constructing Tables 2 and 4 are not quite consistent, as would be expected with a slightly different method of eliciting judgments, and so although the order of the options is the same in each, the relation between the results of the two methods is not exactly linear. The essence of SMARTS is the construction of a multi-attribute value function. There are other approaches which rely on such functions: e.g. methods involving other approaches to deriving value scores and weights (such as the Times University Subject Ratings described above), SMARTER (Edwards and Baron, 1994) which avoids the problems of eliciting weights by deriving them from a mathematical algorithm, and the Analytic hierarchy process (AHP) is arguably a value function approach (Belton and Stewart, 2002). We will take SMARTS as representing a common version of this idea which we can compare with PACOCS. It is convenient to distinguish four differences between PACOCS and SMARTS): 1
PACOCS measures value relative to a well-defined base object. This has the 15
potential advantage of making the resulting scale easier to interpret than the SMARTS scale, and of being closer to “common sense” approaches to making decisions. Wood (2003: 155) suggests a similar approach to multiple regression – rescaling the variables so that the zero point corresponds to a meaningful scenario. 2
The aggregate values in SMARTS are interpreted only in relative terms, whereas in PACOCS the final scale of relative advantage is on a scale which is interpreted in absolute terms. This, together with the first point, mean that the final scale in PACOCS has a clearer interpretation.
3
The calculations in SMARTS involve separating the weights from the partial value scores (vi j wj) whereas in PACOCS no such separation is made (∆vi j). (This is likely to be helpful in so far as the weights are notoriously difficult to derive meaningfully – as discussed in the next point.)
4
SMARTS, but not PACOCS, requires the estimation of swing weights. This process requires the decision maker to imagine n + 1 hypothetical objects (the object which is worst on all criteria and one object for each of the swings) and answer questions about their relative value. This is clearly not trivial, and may be expected to cause difficulties. As argued above, each of these differences can be interpreted as being advantages of
PACOCS over SMARTS in many, perhaps most, circumstances.
Comparison with Even Swaps Another possibility is the method of even swaps (Hammond et al, 1998 and 1999). The idea here is to eliminate the necessity to consider criteria (attributes) by “adjusting” another criteria so that all the objects of evaluation appear identical in value from the perspective of each criterion in turn. For example, in Table 1, if Ride bike were to score "good" on Comfort, all modes of transport would be equal from this point of view, so Comfort could be eliminated from consideration as it is now irrelevant to making the decision. This can be achieved by making an "even swap"—e.g. we might imagine that Ride bike costs more to compensate for improving its score on the Comfort criterion to the level of the other modes of transport, so that the Comfort criterion can then be ignored. According to Table 2, the additional value of the other options is £2.00, so the daily cost needs increasing by this amount – as in Table 5. A similar process can be applied to the other criteria, so that the criteria are eliminated one by one 16
until we have a single criterion left and the problem becomes trivial. In our example, if we eliminate all the criteria except Cost, and the same judgments are made as in Table 2, we will end up with costs for each option with the same intervals between them as the final row of Table 2 (if we eliminate each criterion by making up each option to the level of the best on the given criterion, the costs at the end will be £31.02 for Ride bike, and £36.28, £24.00 and £42.50 for the other three options).
Table 5: First even swap Objects of evaluation Attribute
Go by car
Take train “Ride bike”
Cost (per day)
£14.38
£2.50
£2.52 £4.52
£20.00
Good
Good
bad good
Good
Comfort
Take taxis
Kajanus et al (2001) used the method for a strategy selection problem, concluding that "the even swaps method made this difficult task [assessing trade-offs between objectives] more understandable by indicating the concrete changes in objectives." However, the problem is that the options at the end are not the options we start with, as each time a criterion is eliminated, a change needs to be made to some of the objects, and so the objects at the end may have the same relative value, but they will represent very different, and entirely hypothetical, modes of transport. There are important similarities between PACOCS and Even swaps. Both methods start by focusing on intradimensional comparisons, and, as we have seen, if the judgements in Table 2 are used, both give the same results – in this sense the two methods are arithmetically equivalent. We can summarise the differences between PACOCS and Even Swaps as: 1
Unlike PACOCS, Even swaps requires the decision maker to imagine a series of hypothetical objects – e.g. in Table 5 “Ride bike” is in quotes because it is no longer an ordinary bike which is being ridden, but one that is as comfortable as a car. These hypothetical objects may be difficult to imagine with obvious corollaries for the meaningfulness of the analysis.
2
This difficulty leads through to the final conclusion. Although the final result with an Even swaps analysis is on an interpretable scale (cost), the objects being compared have changed, with obvious possible consequences for ease of interpretation. 17
Both differences are, arguably, in favour of PACOCS.
PACOCS applied to itself? It is tempting now to try to use the advantages of PACOCS over SMARTS and Even swaps (listed above) to derive some common scales and so apply PACOCS to the problem of deciding how to decide. As a rough attempt at this we will use the four differences between PACOCS and SMARTS listed above and generalise them slightly as: 1
Clarity of the base (zero point)
2
Interpretability of the final scale
3
Complexity of the process
4
Hypotheticality – the extent to which it is necessary manipulate hypothetical objects mentally. Any of the methods under discussion requires the decision maker to imagine and evaluate hypothetical states: for example, constructing a partial value scale requires the decision maker to imagine and evaluate a series of objects that vary on only one criterion. However, there are serious concerns about people’s ability to do this coherently (Gilbert, 2007), so the extent and difficulty of these hypothetical judgments is likely to be a major issue in determining the usefulness of any of these methods. Intuitively, I feel able to rank the methods on these criteria – PACOCS is better
SMARTS on all four criteria, and better than Even swaps on criteria 1 (Clarity of the base), 2 (Interpretability of the final scale) and 4 (Hypotheticality), but similar to Even swaps on the other criterion – for the reasons discussed above. Dominance arguments mean that, from my perspective, PACOCS is the best method. However, this is just the perspective of one decision maker in one situation. The relative advantages would doubtless be different with different decision makers and different problems. In particular, the argument that PACOCS is less complex is based on the idea that deriving value scores and weights is more complex than deriving a single assessment of the difference in value. This would obviously depend on the context: in some situations splitting the problem up into two parts may be a way of simplifying it, in others it is likely to complicate it. But as the weighting process is often problematic, it is likely that avoiding it will make the process more transparent on some occasions at least. 18
We will also consider some more general difficulties with PACOCS in the next, concluding, section.
Conclusions and counter-arguments We have argued that PACOCS – the idea of comparing each object of evaluation with a base object, and assessing its Pros And Cons On Common Scale(s) – compares favourably with SMARTS and Even swaps in terms of the clarity and interpretability of the final scales, complexity and the necessity to imagine and evaluate hypothetical objects. This makes it likely to be, in the words of Tversky (1969), “simpler and more natural”. This transparency may make assumptions more visible, and so reduce the danger of silly and inappropriate applications. An important aspect of PACOCS is that it does not involve weights. The idea of weighting criteria is a seductive one, but it is unfortunately difficult to make it precise in a clear and simple manner. Instead of having both value scores, and weights to sum up the importance of each attribute in general terms, PACOCS requires the decision maker to assess the pros and cons of specific objects on clearly defined scales. Despite all this, the arithmetical similarities between the three approaches means that SMARTS or Even swaps cannot claim the advantage of being able to model scenarios which cannot be dealt with by PACOCS. PACOCS can be used in any situation where a multi-criteria assessment is needed – e.g. customer evaluations of (and consequent decisions concerning) goods and services from cars to university courses to medical procedures. Instead of a global evaluation on a scale which may be impossible to interpret, a PACOCS analysis should tell the customer the relative advantage of each product or service compared to the base on one or more readily interpretable scales. However, the above discussion draws mainly on a single example. It is well known that contextual factors in decision making are more important than an abstract analysis might suggest, so it is very probable that the relative advantages of different methods would vary with different contexts and decision makers. Additionally, there are some potential difficulties with PACOCS that deserve mention. 1
The first problem concerns the feasibility of making the judgments required for PACOCS. Are the figures in Table 2 really meaningful? However, any other method requires equivalent judgments (e.g. Tables 4 and 5). This problem is not an argument against PACOCS, but an argument about the limits of any method of 19
multiple criteria decision making. 2
Related to this is the question of whether it is really possible to define scales to reduce the decision to a mono-criterion like the bottom row of Table 2. Again, this procedure is common to other methods like SMARTS and Even swaps – the point about PACOCS is that the procedure is so transparent that any “foolishness” (March, 2006) is likely to be more obvious. By clarifying the rationale behind tradeoffs, PACOCS may, on occasions, force users to recognise that there is no satisfactory way of making trade-offs between some criteria.
3
The choice of the base object, and the common scale, may have a substantial impact on the results – different choices may lead to different rankings of the objects. However, this is equally true of SMARTS and Even swaps. In principle, a consistency check could be built in by repeating the analysis with a different bases, or with different common scales.
4
The main argument for PACOCS is that it is a transparent procedure whose rationale is relatively obvious. Sometimes this may not be what is wanted – the attributes of naturalness and interpretability may be negatively related to value. Decision makers may want an authoritative ranking which will enable them to make the decision without too much thought. There is also the possibility that, for a variety of reasons, the producer of the analysis may not want to encourage users to follow the rationale in detail – this may well be the case for the Times University Subject Tables discussed above. In this case, of course, factors facilitating transparency would be evaluated negatively.
5
PACOCS is arguably a procedure which is so obvious, and so close to what an experienced decision analyst may do instinctively, and to other methods such as “pricing out” and cost benefit analysis, that the arguments of this article may appear trivial. However, inexperienced analysts may not do it instinctively, and the equivalences between PACOCS and Even swaps and SMARTS, suggests that if PACOCS is trivial, then so are these other methods. The suggestion of this paper is that if relatively complex methods, such as SMARTS and Even Swaps, have no advantages over a trivial one such as PACOCS, then the trivial method should be adopted and the more complex ones ignored.
20
References Bell, D. E., Raiffa, H., & Tversky, A. (1988). Descriptive, normative and prescriptive interactions in decision making. in D. E. Bell, H. Raiffa, & A. Tversky (editors), Decision making: descriptive, normative and prescriptive interactions, (pp. 9-30). Cambridge: Cambridge University Press. Belton, V., & Stewart, T. J. (2002). Multiple criteria decision analysis: an integrated approach. Boston: Kluwer. Beynon, M., Rasmequan, S., & Russ, S. (2002). A new paradigm for computer-based decision support. Decision Support Systems, 33, 127-142. Bouyssou, D., Marchant, T., Pirlot, M., Perny, P., Tsoukias, A., & Vincke, P. (2000). Evaluation and decision: models a critical perspective. : Kluwer Academic Publishers. Clemen, R. T. (1996). Making hard decisions (2nd ed). Belmont, Calif: Duxbury Press. Davis, Philip J., & Hersh, Reuben. (1981). The mathematical experience. Brighton: Harvester Press. Edwards, W., & Barron, F. H. (1994). SMARTs and SMARTER: improved simple methods for multiattribute utility measurement. Organizational Behavior and Human Decision Processes, 60, 306- 325. Edwards, W., & Newman, J. R. (2000). Multiattribute evaluation. in T. Connolly, H. R. Arkes, & K. R. Hammond (eds), Judgment and decision making: an interdisciplinary reader (2nd edition), (pp. 17-34). Cambridge: Cambridge University Press. Fischer, Gregory W. (1995). Range sensitivity of attribute wieghts in multiattribute value models. Organizational Behavior and Human Decision Processes, 62(3): 252-266. French, Simon, & Xu, Dong-Ling. (2005). Comparison study of multi-attribute decision analytic software. Journal of Multi-Criteria Decision Analysis, 13(2/3): 65-80. Gilbert, Daniel. (2007). Stumbling on happiness. London: Harper Perennial. Hajkowicz, S., & Higgins, A. (2008). A comparison of multiple criteria analysis techniques for water resource management. European Journal of Operational Research, 184, 255-265. Hammond, J. S., Keeney, R. L., & Raiffa, H. (1998, March). Even swaps: a rational method for making trade-offs. Harvard Business Review, 137-150. Hammond, J. S., Keeney, R. L., & Raiffa, H. (1999). Smart choices. Boston: Harvard
21
Journal of the Operational Research Society. (2006). Special issue on problem structuring methods, 57(7). Kajanus, M., Ahola, J., Kurttila, M., & Pesonen, M. (2001). Application of even swaps for strategy selection in a rural enterprise. Management Decision, 39(5), 394-402. Keeney, R. L. (1992). Value-focused thinking: a path to creative decisionmaking. Cambridge, Massachusetts: Harvard University Press. March, J. G. (2006). Rationality, foolishness and adaptive intelligence. Strategic Management Journal, 27, 201-214. Perris, T., & Labib, A. W. (2004). An intelligent system for prioritisation of organ transplant patient waiting lists using fuzzy logic. Journal of the Operational Research Society, 55, 103-115. Poyhonen, Mari; Hamalainen, Raimo P. (2001). On the convergence of multiattribute weighting methods. European Journal of Operational research, 129 (3): 569-585. Sunday Times. (2006). 100 best companies to work for. London: Sunday Times. Times Online (2007). Good University Guide. Retrieved in November 2007 from http://extras.timesonline.co.uk/gug/gooduniversityguide.php . Tversky, A. (1969). Intransitivity of preferences. Psychological Review, 76(1), 31-48. Wood, M. (2002). Maths should not be hard: the case for making academic knowledge more palatable. Higher Education Review, 34(3), 3-19. Wood, M. (2003). Making sense of statistics: a non-mathematical approach. Basingstoke: Palgrave. Xu, D., & Yang, J. (2003). Intelligent decision system for self- assessment. Journal of Multi-Criteria Decision Analysis, 12(1), 43-60.
22
