Notes on discrepancy in the pairwise comparisons method

Notes on discrepancy in the pairwise comparisons method Konrad Kułakowski Department of Applied Computer Science, AGH University of Science and Technology Al. Mickiewicza 30, 30-059 Cracow, Poland

arXiv:1312.2986v2 [cs.DM] 15 Jan 2014

[email protected]

Abstract. The pairwise comparisons method is a convenient tool used when the relative order among different concepts (alternatives) needs to be determined. One popular implementation of the method is based on solving an eigenvalue problem for the pairwise comparisons matrix. In such cases the ranking result the principal eigenvector of the pairwise comparison matrix is adopted, whilst the eigenvalue is used to determine the index of inconsistency. Studies on the eigenvalue based inconsistency index have led the author to propose a more general index of discrepancy. This describes how far the input to the ranking procedure is from the ranking result. It may help to detect, localize and quantify possible problems related to the input data and the ranking procedure. Some of its properties - those related to the eigenvalue based inconsistency index and the condition of order preservation - are presented as appropriate theorems.

1

Introduction

The origins of pairwise comparisons (herein abbreviated as PC) date back to the thirteenth century [5]. The contemporary form of the method owes to Fechner [7], Thurstone [28] and Saaty [24]. The latter proposed the Analytic Hierarchy Process (AHP) extension to the PC theory, which provides useful methods for dealing with a large number of criteria. In its early stages the PC method was a voting method [5]. Later it was used in psychometrics and psychophysics [28]. Over time, it began to be used in decision theory [25], economics [23], and other fields. The utility of the method has been confirmed by numerous examples [29,12,20,27]. Despite its long existence it is still an interesting subject for researchers. Some of its aspects still raise vigorous discussions [6,2,1] and prompt researchers to enquire further into this area. Example of such exploration are the Rough Set approach [11], fuzzy PC relation handling [21,8,30], incomplete PC relation [3,9,16], data inconsistency reduction [17], non-numerical rankings [14], nonreciprocal PC relation properties [10], rankings with the reference set of alternatives [18,19] and others. A more thorough discussion of the PC method can be found in [26,13].

2 2.1

Preliminaries Pairwise comparisons method

Central to the PC method is a PC matrix M = [m i j ], where m i j ∈ R+ and i , j ∈ {1, . . . , n }, that df

expresses a quantitative relation R over the finite set of concepts C = {c i ∈ C ∧ i ∈ {1, . . . , n }}. The set C is a non empty universe of concepts and R(c i , c j ) = m i j , R(c j , c i ) = m j i . The values m i j and m j i are interpreted as the relative importance, value or quality indicators of concepts c i and c j , so that according to the best knowledge of experts c i = m i j c j should hold. Definition 1. A matrix M is said to be reciprocal if ∀i , j ∈ {1, . . . , n } : m i j = be consistent if ∀i , j , k ∈ {1, . . . , n } : m i j · m j k · m k i = 1.

1 mji

and M is said to

Since the knowledge stored in the PC matrix usually comes from experts in the field of R, it may results in inaccuracy. In such a case it may be that there exists a certain triad of values m i j , m j k , m k i from M for which m i j · m j k · m k i 6= 1. In other words, different ways of estimating concept value may lead to different results. This observation gave rise to the concept of an inconsistency index describing how far the matrix M is inconsistent. There are a number of inconsistency indexes, including Eigenvecor Method [24], Least Squares Method, Chi Squares Method [4], Koczkodaj’s distance based inconsistency index [15] and others. Two of them are defined below. Definition 2. The eigenvalue based consistency index (Saaty’s Index) of n × n reciprocal matrix M is equal to: λmax − n S (M ) = (1) n −1 where λmax is the principal eigenvalue of M . Definition 3. Koczkodaj’s inconsistency index K of n × n and (n > 2) reciprocal matrix M is equal to: ¨ ¨ «« m i j m i k m k j K (M ) = max min 1 − , 1− (2) i ,j ,k ∈{1,...,n } m m m ik

kj

ij

where i , j , k = 1, . . . , n and i 6= j ∧ j 6= k ∧ i 6= k .

The result of the pairwise comparisons method is ranking - a function that assigns values to the concepts. Formally, it can be defined as follows. Definition 4. The ranking function for C (the ranking of C ) is a function µ : C → R+ that assigns to every concept from C ⊂ C a positive value from R+ . In other words, µ(c ) represents the ranking value for c ∈ C . The µ function is usually written in T df  the form of a vector of weights µ = µ(c 1 ), . . . µ(c n ) . One of the popular methods of obtaining the vector µ is to calculate the principal eigenvector µm a x of M (i.e. the vector associated with the principal eigenvalue of M ) and rescale them so that the sum of its elements is 1, i.e. 

µmax (c n ) µmax (c 1 ) ,... , µev = s ev s ev

T

where s ev =

n X

µmax (c i )

(3)

i =1

where µev - the ranking function, µmax - the principal eigenvector of M . Due to the PerronFrobenius theorem [22,24] one exists, because a real square matrix with the positive entries has a unique largest real eigenvalue such that the associated eigenvector has strictly positive components.

3

Eigenvalue heuristics

According to the PC approach m i l (an entry of M ) should express the relative value of c i ∈ C with respect to c l ∈ C . Therefore one would expect that µ(c i )/µ(c l ) = m i l , i.e. µ(c i ) = m i l µ(c l ) or conversely m l i µ(c i ) = µ(c l ). In particular, it is desirable that m l i µ(c i ) = µ(c l ) = m l j µ(c j )

(4)

for every c i , c j , c l ∈ C . Unfortunately due to possible data inconsistency this may not be possible, i.e. it may be the case that m l i µ(c i ) 6= m l j µ(c j ). Therefore the question arises of what µ(c l ) should be? Since the values m l i µ(c i ) for i = 1, . . . , n can vary from each other the natural (and probably one of the most straightforward) proposal is to adopt its arithmetic mean as the desired candidate for µ(c l ). This leads to the equation: m l 1 µ(c 1 ) + . . . + m l n µ(c n ) = n · µ(c l )

(5)

which expresses the wish that µ(c l ) should be a compromise between all its putative values. A natural question is whether it is possible to achieve such a compromise for every l = 1, . . . , n . In other words, whether it is possible to solve the following equation system: m 11 µ(c 1 ) + . . . + m 1n µ(c n ) = n · µ(c 1 ) ........................................... m n 1 µ(c 1 ) + . . . + m n n µ(c n ) = n · µ(c n )

(6)

This leads to the question of the solution of the following matrix equation: Mµ = nµ

(7)

Of course, when the matrix M is consistent, i.e. every m l i µ(c i ) equals any other value m l j µ(c j ) in a row1 then the equation 7 holds. In general the solution of an equation of this form (7) is the eigenvector of M , whilst n is replaced by λ - M ’s eigenvalue. M µ = λµ

(8)

There might be many eigenvectors and eigenvalues of M . However, when M is positive, real and reciprocal it has at least one positive real eigenvalue associated with the positive and real eigenvector [22,24]. Let λmax be the real, largest, positive eigenvalue of M and µmax be the associated eigenvector. The pairwise comparisons method adopts µmax as the solution of (8) believing that the value µmax (c l ) for every l = 1, . . . , n is a good approximation of the mean Pn 1 m l i µmax (c i ) (compare with 5). i =1 n

4

The problem with inconsistency

The question arises of how good this approximation is? How does it depend on the level of inconsistency? Let us consider the distance ∆l defined as: n X 1 (9) ∆l = µmax (c l ) − m l i µmax (c i ) n i =1

1

In other words due to (8) n − λ 1 max ∆l = µmax (c l ) − λmax µmax (c l ) = µmax (c l ) n n

(10)

Note that M is consistent, thus in particular according to (Def. 1) m l i m i j m j l = 1. Hence, due to the reciprocity m l i m j l = m j i , i.e. m l i m j l = µ(c j )/µ(c i ). Therefore, when M is consistent m l i µ(c i ) = 1/m j l µ(c j ), i.e. m l i µ(c i ) = m l j µ(c j ).

and since λmax ≥ n (see [24]) ∆l = µmax (c l )

λmax − n n

(11)

The equation (11) could also be written (see Def. 1) as: ∆l n = S (M ) n − 1 µmax (c l )

(12)

In other words the distance ∆l depends on the inconsistency index S (M ) and of course if S (M ) = 0 then also ∆l = 0. Unfortunately every change of ∆l may entail a change of µmax (c l ). Since, “there is no easy way to study the sensitivity of the eigenvector µmax to errors in M ” [24] it is hard to judge whether the small improvement in inconsistency S (M ) always translates into a decrease of ∆l . However, according to the eigenvalue heuristics, one can expect that the essential decrease of S (M ) should entail a decrease rather than increase of ∆l . us look more closely to the Eq 12. Since λmax ≥ n (see [24]) implies that µmax (c l ) ≤ PLet n 1 m l i µmax (c i ) the Equation (12) could be written as: i =1 n n

hence

Let us denote:

€ 1 Pn n

i =1

m l i µmax (c i ) − µmax (c l )

(n − 1)µmax (c l )

Š

= S (M )



 n X 1 m l i · µmax (c i )   − 1 = S (M ) (n − 1) i =1,i 6=l µmax (c l ) df

κ(l , i ) = m l i

1 µ(c i ) µ(c i ) = µ(c l ) m i l µ(c l )

(13)

(14)

(15)

If the ranking µmax were ideal, i.e. if each expert judgment perfectly corresponded to the ranking results, then every m i l would equal the ratio µ(c i )/µ(c l ). In such a case every κ(l , i ) would equal to 1. Otherwise, when the ranking is imperfect the values m i l and µ(c i )/µ(c l ) may vary. In other words κ(l , i ) is in fact the assessment accuracy determinant. That is because it determines how much m i l - the particular expert judgment (an expert expectation as regards the pair c i , c l ) differs from µ(c i )/µ(c l ) - the ranking result. The relationship between κ(l , i ) and S (M ) (adopting µmax - the eigenvector of M as the ranking function) could be written as follows: n X 1 (κ(l , i ) − 1) = S (M ) (n − 1) i =1,i 6=l

(16)

In other words the given value of the inconsistency index S (M ) guarantees that the arithmetic mean of the difference between assessment accuracy determinants and one, i.e. κ(i , l )−1 equals S (M ). Unfortunately, this estimation leaves a lot of freedom to individual values of κ(i , l ). When some are relatively high, others could be small. It is easy to see that κ(l , i ) = 1/κ(i ,l ). For example if some κ(l , i ) = 2 then κ(i , l ) = 0.5. In fact both of these values carry the same information, which is: the ranking result for the pair (c l , c i ) differs twice from the expert judgement. I.e. one concept got 100% better score than they should.

Let us define the local error E (l , i ) as: df

E (l , i ) = max{κ(l , i ) − 1, 1/κ(l ,i ) − 1}

(17)

The value E (l , i )·100% reflects local differences between ranking results and given expert judgements. Let us consider an example where M be the following PC matrix: 

 1 0.24 1.1 0.91  1   0.24 1 1.28 1.1  M = 1  1  1.1 1.28 1 0.5  1 1 2 1 0.91 1.1

(18)

As it is easy to check S (M ) = 0.079. Thus, it is slightly below the usual limit of acceptability for the matrix 4 × 4 (which is 0.08 [25]), but it is still acceptable. The corresponding matrices κ(M ) = [κ(l , i )] and E (M ) · 100% = [E (l , i ) · 100%] with respect to µmax are as follows:    1 0.546 1.192 1.499 0% 83% 19% 50%      1.831 1 0.61 0.797   83% 0% 64% 25%  κ(M ) =   E (M ) · 100% =    0.839 1.639 1 0.76   19% 64% 0% 32%  0.667 1.255 1.316 1 50% 25% 32% 0% 

(19)

In other words, although the inconsistency S (M ) is reasonably small, i.e. it is below the 8% threshold, the local discrepancies between the ranking result and the expert judgement reach 80%. The matrices κ(M ) and E (M ) illustrate the distribution of the discrepancy between M and µ. They could be considered as maps of discrepancy for M and µ, and allow to discover where the highest discrepancy is, hence, where the expert judgement (or the ranking function) could be improved.

5

The ranking discrepancy index

The situation, in which the expert judgment differs by 80% from the final result (an expert made a mistake in judgment by 80%) and the result remains valid, is not comfortable. Therefore, in order to reduce (to limit) the local discrepancies it is reasonable to introduce the concept of the ranking discrepancy index. Definition 5. Let the ranking discrepancy index D(M , µ) for the pairwise comparisons matrix M , and the ranking µ, be the maximal value of E (l , i ) for l , i = 1, . . . , n , i.e. df

D(M , µ) =



max E (l , i )

l ,i =1,...,n



(20)

Thus, a certain value of the ranking discrepancy index D(M , µ) ≤ δ provides a guarantee that the maximal discrepancy between a single assessment of an expert and the comparison of corresponding results will not be greater than δ. The ranking discrepancy D(M , µ) translates directly into the inconsistency S (M ). The relationship can be expressed as the following theorem.

Theorem 1. For every pairwise comparisons matrix M and the eigenvector based ranking µm a x holds that: D(M , µmax ) ≤ δ ⇒ S (M ) ≤ δ

(21)

Proof. Since D(M , µ) ≤ δ, thus according to the definition 5, every E (l , i ) ≤ δ for l , i = 1, . . . , n . Thus, due to definition of E (see 17), holds that κ(l , i ) − 1 ≤ δ for every l , i ∈ {1, . . . , n }. In particular for any l ∈ {1, . . . , n } it is true that: n X

(κ(l , i ) − 1) ≤ (n − 1)δ

(22)

n X 1 (κ(l , i ) − 1) ≤ δ (n − 1) i =1,i 6=l

(23)

i =1,i 6=l

hence

which, in the light of (16) satisfies the assertion S (M ) ≤ δ. Hence, besides the fact that the ranking discrepancy index D(M , µmax ) detects and limits the worst case discrepancy between a single expert judgement and the ranking result, it also provides a guarantee in the original sense proposed by Saaty [25]. Therefore wherever the inconsistency index S (M ) has so far been used, D(M , µmax ) might be used instead. Provided of course, that D(M , µmax ) is sufficiently small. In return, in addition to the requirements of the level of inconsistency, the users receive a guarantee of even discrepancy distribution.

6

The ranking discrepancy and the conditions of order preservation

In [1] Bana e Costa and Vansnick formulate two postulates (conditions of order preservations) as regards the meaning of an eigenvalue based ranking result. The first one, ordinal, the preservation of order preference condition (POP) claims that the ranking result in relation to the given pair of concepts (c i , c j ) should not break with the expert judgement. In other words for pair of concepts c 1 , c 2 ∈ C such that c 1 dominates c 2 i.e. m 1,2 > 1 should hold that: µ(c 1 ) > µ(c 2 )

(24)

The second one, cardinal, the preservation of order of intensity of preference condition (POIP), stipulates that if c 1 dominates c 2 , more than c 3 dominates c 4 (for c 1 , . . . , c 4 ∈ C ), i.e. if additionally m 3,4 > 1 and m 1,2 > m 3,4 then also µ(c 1 ) µ(c 3 ) > (25) µ(c 2 ) µ(c 4 ) The straightforward relationship between the ranking discrepancy index D(M , µ) and POP and POIP could be expressed in the form of the following two assertions. Theorem 2. For every pairwise comparisons matrix M expressing the quantitative relationships R between concepts c 1 , . . . , c n ∈ C , and the ranking µ, the order preference condition is preserved i.e. m i j > 1 implies µ(c i ) > µ(c j ) (26) if wherever D(M , µ) < δ then also m i j ≥ δ + 1.

Proof. Since D(M , µ) < δ, then according to the definition 5, every E (p,q ) < δ for p,q = 1, . . . , n . In particular E (i , j ) < δ, hence also κ(i , j ) − 1 < δ. Therefore, due to the definition of κ (15) it is true that 1 µ(c j ) µ(c j ) δ + 1

(28)

and due to the reciprocity mi j µ(c i ) > µ(c j ) δ + 1

(29)

Therefore the ratio µ(c i )/µ(c j ) is strictly greater than one if only m i j /δ+1 ≥ 1. In other words the only requirement in addition to D(M , µ) < δ needed to meet the POP is m i j ≥ δ + 1. The above theorem easily translates into an algorithm that allows us to decide whether the pairwise comparison matrix M and the ranking µ are POP-safe, i.e. whether the POP condition will never be violated for this pair. Let us note that if we adopt a weak inequality as the upper bound of the ranking discrepancy index i.e. D(M , µ) ≤ δ, then to meet the POP the strong inequality m i j > δ + 1 is needed. Thus, assuming that δ = D(M , µ) is known, all the ratios greater than one i.e. m i j > 1 need to be examined to determine whether they are also greater than δ + 1. If so, M is POP-safe, which means that POP is not violated. The relationship between POIP and D(M , µ) also can be expressed in the form of assertion. Theorem 3. For every pairwise comparisons matrix M expressing the quantitative relationships R between concepts c 1 , . . . , c n ∈ C , and the ranking µ, the order of intensity of preference condition is preserved i.e. µ(c i ) µ(c k ) > (30) m i j > m k l > 1 implies µ(c j ) µ(c l ) if wherever D(M , µ) < δ then also m i j /m k l ≥ (δ + 1)2 Proof. Since D(M , µ) < δ, then according to the definition 5, every E (p,q ) < δ for p,q = 1, . . . , n . In particular E (i , j ) < δ, hence also κ(i , j ) − 1 < δ and κ(l , k ) − 1 < δ. Thus, following the same reasoning as in Theorem 2 (27, 28 and 29) we obtain that mi j µ(c i ) µ(c l ) ml k > and > µ(c j ) δ + 1 µ(c k ) δ + 1

(31)

hence due to the reciprocity, mi j µ(c k ) µ(c i ) > and < m k l (δ + 1) µ(c j ) δ + 1 µ(c l )

(32)

Therefore, dividing the left inequality by the right inequality leads to the formula µ(c i ) µ(c j ) µ(c k ) µ(c l )

>

mi j δ+1

m k l (δ + 1)

(33)

Therefore, the ratio µ(c i )/µ(c j )/µ(c k )/µ(c l ) is greater than 1 if m i j /(δ+1)/m k l (δ+1) is not smaller than 1. In other words the truth of the following inequality: mi j

≥ (δ + 1)2

(34)

µ(c i ) µ(c k ) > µ(c j ) µ(c l )

(35)

mkl implies that

which is the desired assertion. Similar as before, to hold the above theorem it is enough for the weak inequality D(M , µ) ≤ δ and the strong inequality m i j /m k l > (δ + 1)2 to hold. Thus, for the practical verification of whether the POIP is violated, the condition m i j /m k l > (δ + 1)2 needs to be examined for every pair m i j , m k l that meets the requirements of the theorem.

7

Discussion and summary

Although the ranking discrepancy index D(M , µ) (Sec. 5) has been defined in the context of eigenvalue heuristics, it is not tied to it. In fact it could be useful for any pair of the PC matrix M and the ranking µ. The only, but crucial, assumption is that µ attempts to reflect the experts’ judgments given as M . It differs from the notion of inconsistency indices, since it needs both the pairwise comparisons matrix M and the result vector µ. It attempts to estimate the level of discrepancy between expert judgements (matrix M ) and the result of the ranking procedure (vector µ). Inconsistency indices (as e.g. Koczkodaj’s distance based inconsistency index [15]) decide on the level of discrepancy among different expert judgements and do not consider the ranking procedure. Despite these differences in some cases D(M , µ) may play a similar role to the index of inconsistency. This happens in the case of the eigenvalue based pairwise comparison method. When the principal eigenvector µmax is adopted as the ranking function, the value D(M , µmax ) ensures that the well known inconsistency criterion S (M ) is preserved (Theorem 1). Of course, it also limits local discrepancies between experts judgements and the ranking result. In their work Bana e Costa and Vansnick [1] formulated two conditions whose fulfillment makes the ranking result indisputable. Therefore, in practice, meeting these two conditions may translate into a significant reduction in the number of appeals against the results of the ranking procedure. Hence, in addition to intangible benefits such as providing the ranking participants a sense of justice, meeting the POP and POIP conditions may contribute to the reduction of costs associated with the carrying out the evaluation procedure. The ranking discrepancy index D(M , µ) helps to fulfill the Bana e Costa and Vansninck postulate. It seems to be a missing link between conditions of order preservations [1] and S (M ) [24]. The value D(M , µ) directly translates to the requirements for the matrix M , so that the smaller D(M , µ) the greater the chance that the POP and POIP conditions are met. This study addresses an important problem of discrepancies between expert judgments and ranking results that may appear in the pairwise comparison method. A discrepancy index has been defined. Its relationship with the eigenvalue based inconsistency index and POP and POIP [1] postulates have been shown. An open question remains concerning its relation to other inconsistency indices such as Koczkodaj’s distance based inconsistency index [15].

Acknowledgements I would like to thank Dr Jacek Szybowski and Prof. Antoni Lig˛eza for reading the first version of this work. Special thanks are due to Dan Swain for his editorial help.

References 1. C. A. Bana e Costa and J. Vansnick. A critical analysis of the eigenvalue method used to derive priorities in AHP. European Journal of Operational Research, 187(3):1422–1428, June 2008. 2. J. Barzilai and B. Golany. AHP rank reversal, normalization and aggregation rules. INFOR - Information Systems and Operational Research, 32(2):57–64, 1994. 3. S. Bozóki, J. Fülöp, and L. Rónyai. On optimal completion of incomplete pairwise comparison matrices. Mathematical and Computer Modelling, 52(1–2):318 – 333, 2010. 4. S. Bozóki and T. Rapcsak. On Saaty’s and Koczkodaj’s inconsistencies of pairwise comparison matrices. Journal of Global Optimization, 42(2):157–175, 2008. 5. J. M. Colomer. Ramon Llull: from ‘Ars electionis’ to social choice theory. Social Choice and Welfare, 40(2):317–328, October 2011. 6. J. S. Dyer. Remarks on the analytic hierarchy process. Management Science, 36(3):249–258, 1990. 7. G. T. Fechner. Elements of psychophysics, volume 1. Holt, Rinehart and Winston, New York, 1966. 8. M. Fedrizzi and M. Brunelli. On the priority vector associated with a reciprocal relation and a pairwise comparison matrix. Journal of Soft Computing, 14(6):639–645, January 2010. 9. M. Fedrizzi and S. Giove. Incomplete pairwise comparison and consistency optimization. European Journal of Operational Research, 183(1):303–313, 2007. 10. J. Fülöp, W. W. Koczkodaj, and S. J. Szarek. On some convexity properties of the least squares method for pairwise comparisons matrices without the reciprocity condition. J. Global Optimization, 54(4):689–706, 2012. ´ 11. S. Greco, B. Matarazzo, and R. Słowinski. Dominance-based rough set approach on pairwise comparison tables to decision involving multiple decision makers. In JingTao Yao, Sheela Ramanna, Guoyin Wang, and Zbigniew Suraj, editors, Rough Sets and Knowledge Technology, volume 6954 of Lecture Notes in Computer Science, pages 126–135. Springer Berlin Heidelberg, 2011. 12. William Ho. Integrated analytic hierarchy process and its applications - A literature review. European Journal of Operational Research, 186(1):18–18, March 2008. 13. A. Ishizaka and A. Labib. Analytic hierarchy process and expert choice: Benefits and limitations. OR Insight, 22(4):201–220, 2009. 14. R Janicki and Y. Zhai. On a pairwise comparison-based consistent non-numerical ranking. Logic Journal of the IGPL, 20(4):667–676, 2012. 15. W. W. Koczkodaj. A new definition of consistency of pairwise comparisons. Math. Comput. Model., 18(7):79–84, October 1993. 16. W. W. Koczkodaj, M. W. Herman, and M. Orlowski. Managing Null Entries in Pairwise Comparisons. Knowledge and Information Systems, 1(1):119–125, 1999. 17. W. W. Koczkodaj and S. J. Szarek. On distance-based inconsistency reduction algorithms for pairwise comparisons. Logic Journal of the IGPL, 18(6):859–869, October 2010. 18. K. Kułakowski. A heuristic rating estimation algorithm for the pairwise comparisons method. Central European Journal of Operations Research, pages 1–17, 2013. 19. K. Kułakowski. Heuristic Rating Estimation Approach to The Pairwise Comparisons Method. CoRR, (accepted for publication in Fundamenta Informaticae), abs/1309.0386, 2013. 20. M. J. Liberatore and R. L. Nydick. The analytic hierarchy process in medical and health care decision making: A literature review. European Journal of Operational Research, 189(1):14–14, August 2008. 21. L. Mikhailov. Deriving priorities from fuzzy pairwise comparison judgements. Fuzzy Sets and Systems, 134(3):365–385, March 2003. 22. O. Perron. Zur Theorie der Matrices. Mathematische Annalen, 64(2):248–263, June 1907. 23. G. L. Peterson and T. C. Brown. Economic valuation by the method of paired comparison, with emphasis on evaluation of the transitivity axiom. Land Economics, pages 240–261, 1998. 24. T. L. Saaty. A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology, 15(3):234 – 281, 1977. 25. T. L. Saaty. The analytic hierarchy and analytic network processes for the measurement of intangible criteria and for decision-making. In Multiple Criteria Decision Analysis: State of the Art Surveys, volume 78 of International Series in Operations Research and Management Science, pages 345–405. Springer New York, 2005.

26. J. E. Smith and D. Von Winterfeldt. Anniversary article: decision analysis in management science. Management Science, 50(5):561–574, 2004. 27. N. Subramanian and R. Ramanathan. A review of applications of Analytic Hierarchy Process in operations management. International Journal of Production Economics, 138(2):215–241, August 2012. 28. L. L. Thurstone. A law of comparative judgment, reprint of an original work published in 1927. Psychological Review, 101:266–270, 1994. 29. O. S. Vaidya and S. Kumar. Analytic hierarchy process: An overview of applications. European Journal of Operational Research, 169(1):1–29, February 2006. 30. K. K. F. Yuen. Fuzzy cognitive network process: Comparison with fuzzy analytic hierarchy process in new product development strategy. Fuzzy Systems, IEEE Transactions on, PP(99):1–1, 2013.