Towards a web-based collaborative weighting method in project

Towards a web-based collaborative weighting method in project Bernard Yannou Laboratoire Génie Industriel Ecole Centrale Paris Grande Voie des Vignes, 92295, Châtenay-Malabry, France [email protected] Abstract-- For the purpose of a product design or a project in general, weighting a set of comparable criteria has been proven to be of utmost importance (e.g. weighting product functions in value analysis - VA -, and allocating a budget in a Design-ToCost project). Moreover the weighting problem is related to basic properties in the field of Multi-Criteria Decision Analysis (MCDA) through the notions of ordinal transitivity and rationality in the designers' mind. How should designers or project agents decide in the presence of uncertainties and different preference logics in a quick and efficient way (reactivity)? One of the weighting methods currently used in VA studies and in MCDA practices (through AHP approaches for example) is the pairwise comparison method. Recent improvements have been made in this field for a practical use in projects, with Limayem’s work which permits to obtain more precise estimations in the presence of uncertainties (fuzzy logic and statistical approaches), to take into account several decision makers at a time without requiring them to master the whole expertise, and to provide them with hints to locate parts of their judgment which do not seem consistent with the main part in order to improve it. Such properties let us hope that it should be possible to propose a web-based collaborative decision making method based upon an asynchronous and remote mode. After a presentation of Limayem's advances in the research on pairwise comparison, we describe a flexible framework of web-based vote protocols to yield a weight consensus.

Keywords: Pairwise comparison, collaborative decision making, group decision support, vote protocols

objectives of a project or of a company strategy, or the occurrence probabilities of events, risks… For example, it occurs in a product design project with the weighting stage of the expected functions of the product. This weighting is carried out in the sense of the participation impulse to the purchase act for the targeted market segment. This weighting vector is useful for the designers to represent the importance of functions for the end-users roughly: it consequently directs them to focus the efforts of redesign or innovation at best with regards to the clients’ expectations. The Value Analysis approach gives some tools to lead both a technical and economical optimisation by attempting not to invest too much in functions of lesser importance for the client. In the field of optimisation, the elements of the objective functions or the penalties must also be weighted. B. The difficulties one may expect Faced with the problem of weighting, decision makers are often embarrassed. It is actually rather difficult to straightforwardly result in a distribution of one hundred percent on a set of elements, even when accustomed to using a basic scoring method. The following common issues appear: -

I. THE WEIGHTING ACTIVITY IN PROJECT

A. Weighting and decision making Confronted to a number of alternative solutions, or elements, certain decisions need to express a vector of weights. This is the case in politics, as well as in the social or economical domains, and in those of engineering and company management (product design and project management for example). Indeed, when a resource has to be split, it is required to fix the size of each subpart (element), according to the importance of the elements. This situation occurs when a budget has to be split into different activities, different managers, or different objectives as in Design-To-Cost, or in company management (e.g.: investments in quality actions). This is also the case when time has to be allocated to activities. Starting from the total duration of the project, the time partition must be chosen at best for the project team. Sometimes there is no resource to allocate ; one rather has to assess the importance of some properties such as the

-

-

-

Lack of information – Imprecision – Uncertainty. When the decision must be made, the appropriate, precise, certain (validated) information lacks, or the decision makers themselves are not certain of their preferences. Too many elements to compare. When there are more than 6 elements to order, compare, or weight, it gets too confusing for a human being. More confusion is brought when several decision makers are involved in the decision making process. Discordant opinions in a group. When the decision making is made by a participative group, each decision maker is allowed to set a personal order or weight to the elements. Indeed, each decision maker has his/her own system of preferences, according to his/her interests inside the company, his/her personal culture and sensitivity. An important issue is the following: Should the different opinions be weighted and how? Or should one reach a one and only opinion in a more consensual way? Impossible assessments. Sometimes it is impossible to compare certain elements with others. It may be due to the fact that such a comparison is realized through several criteria and none of the compared elements is dominant as far as all criteria are concerned. In such a

-

case the implicit aggregation of several criteria into the criterion of importance is not simple. In a monocriterion case, the difficulty to compare may occur by lack of knowledge. Some weight results arbitrarily precise. Traditionally, when a given decision making process results in two close weight values for two elements, an executive may feel quite confident. But he/she should not trust such a tight result, especially if the weighting process aimed to select the highest weighted element. Indeed, the weighting vector partly obtained in an intuitive manner is affected by a certain degree of uncertainty. Once this uncertainty is taken into account, there is no certitude that the two previous elements will be discriminated, leading to a so-called indifference between the elements. Such cases have to be identified in order to consider more strategic criteria within the decision making process. C. Introduction to Pairwise Comparison methods

How can one obtain a sound weight vector, for a sole decision maker as well as for a group, in an uncertain decision context, and while respecting the different individual opinions in case of a participative group as much as possible? More than 20 years ago, Saaty [16; 17] proposed the Analytic Hierarchy Process (AHP) method to decompose the multi criteria weighting of a set of elements into several steps of mono criterion weighting. Each step is then skipped with the use of a Pairwise Comparison (PC) method. Saaty himself proposed the Eigenvector Pairwise Comparison Method. The principle of Pairwise Comparison methods was introduced by Thurstone in 1927 [18]. These methods permit to simplify the distribution of 100% of importance into n elements (mono criterion case). The decision makers may be one or more and are denoted DMs here. The PC principle consists in comparing successively the relative importance of element i and element j, by means of assessment of their ratio. An elementary comparison is noted: c ij ≈ p i p j , i, j = 1...n . These comparisons are

This is why the system is a priori inconsistent. Then, the different methods express different logics of compromise. The methods often depend on the fact that the matrix M has the property of reciprocity or not. This property is verified whenever cij=1/cji; i,j = 1..n, and consequently when cii=1. Most of the time, such is the case when the order of comparison between element i and element j does not matter and when there is no blind test (i.e. an element is recognized equal to itself). The number of comparisons becomes n(n-1)/2 instead of n². Some methods allow each DM to express his/her own opinion by comparison or to omit an opinion on a comparison. Matrix M then acquires a third dimension : M=(cijk), with k=0,…,dij, dij being the number of opinions by comparison.

-

-

Despite this variety of methods and the resulting variety of results, these methods must converge to the same result when the matrix is consistent. A matrix turns out to be consistent when all the comparisons may be expressed as the actual ratios of two weights. There is no chance that in practice, such a property is verified as the comparisons are evaluated independently by the DMs all along the matrix filling process. An inconsistent matrix does not comply with the cardinal transitivity expressed by cih⋅chj= cij; pour tous i,j,h = 1..n. As a consequence, whenever a consistent matrix M is artificially built from a given weight vector

( p1 ,..., pi ,..., p n ) , that is

PC

method

. c1n   . .  , denoted M = c ij . .   . c nn 

( )

must

predict the same weight vector as a result. No bias in the initial judgments is introduced. The author observed that some PC methods used in the practice of Value Analysis methodology did not comply with this basic property, which is a dangerous practice. Let us take the comparison matrix of figure 1 as an example. This matrix is consistent because the comparisons are compatible with e3=2.e2=6.e1. In considering 6 parts for e3, 3 parts for e2, and 1 part for e1, the result is logically: e1=10%, e2=30%, e3=60%, whatever the PC method is.

e1 (1)

A PC method starts from a matrix M filled by the decision group to foresee a weight vector which will be the most representative of the composition of the comparisons. This weight vector is, in general, not unique and a variety of methods may coexist since: -

The n² assessed comparisons may be considered as n² equations and, as the number of unknown variables is n for the weights pi to be determined, the number of equations is too large to be mathematically solvable.

)

( p1 ,..., pi ,..., p n )

gathered in a square comparison matrix M :

 c11 .  .  . M = . cij  c  n1 .

(

M c = pi / p j , any acceptable

e2

e3

e1

1 13 16

e2

3

1

12

e3

6

2

1

Figure 1: Example of a consistent comparison matrix D. Pairwise Comparison and the DM rationality The PC methods are tightly tied with the important issue of a DM rationality or of the rationality of a decision group. The rationality of a DM is defined by his/her ability to express a set of preferences between pairs of elements, which is consistent with the order relation. If A is preferred to B is denoted by A>B, the following case must not appear: A>B>C>A. Such a circular reference is a violation of the

ordinal transitivity. This ordinal transitivity can easily be checked from the comparison matrix. The property of DM rationality is a required condition for developing models in the utility theory (cf. [7; 8])., for instance. But this is not a sufficient condition. As Arrow’s impossibility theorem (cf. [1]) highlights that, even with rational DMs, the result of a participative project is likely to be irrational anyway. As Herman and Koczkodaj [9] suggested, the PC methods contribute to lower the group’s irrationality, since the DMs are compelled to exchange their points of view in a participative way and to result in a consensual comparison valuation, or to accept the result proposed by the method. Another important issue is the non-respect of the cardinal transitivity. A cardinal transitivity implies an ordinal transitivity which is not true conversely. A number of scientific works have dealt with the assessment of this cardinal transitivity. Nevertheless, there exists an epistemological debate (see Lootsma [15] and Finan and Hurley [6]) on the fact that the decision group should be asked to ensure the ordinal transitivity and to improve somewhat the cardinal transitivity. As this debate is not over, one decided in this work to only propose to the DMs several degrees of consistency for assessing cardinal transitivity, as an incentive for them to change their opinions voluntarily.

II. A BASIC STATE-OF-THE-ART FOR PC METHODS A. The Least Square Logarithmic Regression (LSLR) method One of the most popular PC method, because of its flexibility, is the Least Square Logarithmic Regression (LSLR) method. De Graan [4] and Lootsma [14] authorized multiple opinions or none by comparisons in the case of a reciprocal matrix. Let us note the comparison cijk, with k the DM index varying from 1 to dij. They mean to minimize the least square logarithmic distance: n

n

d ij

∑ ∑ ∑ (log(cijk ) − (log( pi ) − log( p j )))

2

(2)

i =1 j =i +1 k =1

This system leads to a linear system resolution with (n-1) equations and (n-1) unknown variables θ i , i = 1,..., n-1 : n

n−1

n

j ≠i

j ≠i

j ≠i k =1

d

θ i ⋅ ∑dij − ∑dij ⋅ θ j = ∑∑log(cijk ), i = 1,..., n-1 ; θn = 1 The final result is given by: p i =

exp(θ i )

∑ j =1θ j n

(3)

, i = 1,...,n (4)

The matrix example of figure 1 is modified to set the matrix slightly inconsistent in figure 2.

e1 e2 e3

e1 e2 e3 1 13 15 3 1 12 5 2 1

e1 10.9% e2 30.9% e3 58.2%

Figure 2: Example of an inconsistent and reciprocal comparison matrix. B. Taking uncertainty into account There are two theories which take the uncertainty during the evaluation of the comparisons into account: fuzzy logic and probabilistic approach. Van Laarhoven and Pedrycz [19] were the first to extend the LSLR approach to fuzzy logic. They considered a comparison cijk as fuzzy triangle (the y-axis being the likelihood), i.e. a triplet (cijkb, cijkm, cijkh), cijkb being the lower bound of the ratio p i / p j that can be envisaged by DM k, cijkh being the upper bound of the ratio p i / p j and cijkm being the most probable value of this ratio. Let us recall that the interest to obtain fuzzy weights is to be able to detect an indifference between two elements and to avoid to illegitimately favor an element versus another. Injecting the fuzzy triplets into the equation set (9), Van Laarhoven and Pedrycz obtain explicit formulas for the weights p i = ( p ib , p im , p ih ) . But their formulas may give some inconsistent results as cijkb>cijkh, and later Boender [2] proposed best formulas. A second attempt with fuzzy logic was carried out by Buckley [3] who used Zadeh’s conventional fuzzy extensions principles. But this approach had two severe limitations. First, the LSLR method cannot be extended as there is no explicit set of equations. Second, the resulting fuzzy numbers for the weights are much broader, leading to the indifference conclusion too often. In a recent Ph. D. dissertation, Limayem [11] and the author proposed numerous advances in developing flexible PC methods taking uncertainty into account, and particularly adapted to a project context. First, they proposed a theoretic generalisation of the LSLR approach and in turn a new fuzzy computation of the equation set with the Fuzzy Weighted Average approach [5]. The resulting method is much more flexible than Boender’s [2]. A second approach is proposed by Limayem and the author. This is a probabilistic approach named MCPC for Monte Carlo Pairwise Comparison. The comparisons are defined by different kinds of probability density functions (pdfs), and a Monte Carlo simulation is performed (hundreds of valued comparison matrices are randomly chosen) before reconstructing the resulting weight pdfs. Previous probabilistic approaches already existed [6; 9; 10] but Limayem and the author developed additional consistency indicators for the use in a project context.

III. TOWARDS A WEB-BASED DECISION TOOL IN GROUPS A. New consistency indicators for assessing the quality of the decision Limayem [11] proposed a set of consistency indicators which: are defined between 0 (total inconsistency) and 100% (consistency), are based on the least square logarithmic distance between the initial comparison matrix M and the consistent matrix reconstructed from the resulting weights Mc : -

∑ (log (cijc )− log (cij ))

2

(5)

are generalized for several opinions by comparison and values different from 1 on the diagonal (blind tests), can be parameterized according to the type of permitted (by the group and its leader) inconsistency (not detailed here, see [11]), result in a global consistency indicator ICglobal for judging the inconsistency of matrix M=(cijk), also result in personal consistency indicators ICk for the DMs k who have expressed a sufficient number of personal opinions to have a corresponding rank of equation set greater or equal to n-1.

-

-

In addition, Limayem and the author proposed a twofold series of consistency improvement tracks (CITs) that can be followed or not by the DMs to revise their judgments. Let us explain the principle of these CITs with one opinion per comparison (dij=1). These tracks consist in indicating which revisions of the comparison would benefit in priority to an improvement of the consistency. For the most promising sources of improvement, comparisons are flagged with the level of priority (1 for the highest priority) and a direction of change (+ for increasing, - for decreasing). We avoided to provide a precise influence factor value in order not to influence the DM too much into necessarily revising his/her judgment. The algorithmic principle to produce these improvement tracks is quite simple (see figure 3) ; it is based on the absolute value of the difference between the two matrices M et Mc, a large difference for a comparison indicating that this comparison is an important source of inconsistency relatively to the other comparisons. M

CP Method Weights

Mc

Difference M-Mc

Comparisons to revise in priority with the direction of change Figure 3: Principle of the consistency improvement tracks (CITs)

In fact, the (m+1) consistency indicators ICglobal and ICk (m being the number of DMs) are accompanied by 2.m sets of

consistency improvement track matrices, i.e. 2 CITs by DM which are: a CIT relatively to his/her matrix Mk for improving at best his/her own ICk consistency, a CIT relatively to his/her matrix Mk for influencing at best the global consistency indicator ICglobal. B. The elementary vote session A weighting session in a group turns out to be rather flexible, instructive (the group learns on itself) and very respectful of every opinion with a series of volunteer revisions of the personal judgment. When all the DMs decide to freeze their judgments, the group has converged towards a consensus. Of course, this consensus, as all consensus, is more or less accepted and the final personal and group consistencies may be more or less satisfactory but the consideration of personal opinions has been carried as far as possible. The revisions may be synchronised to proceed along a series of votes. An elementary vote stage is represented in figure 4. Practically, each DM can decide by him/herself to modify his/her own opinion to move either in the sense of a greater personal consistency or in the sense of a greater synergy with the group. Each DM is free to develop his/her own strategy depending on his/her indicators and on the group’s, but is it not always the case in a company? Tracks for personal / global consistency improvement Personal Ick and weights ( p ,..., p ,..., p ) 1 i n Personal Ick and weights ( p ,..., p ,..., p ) 1 i n Personal Ick and weights ( p ,..., p ,..., p ) 1 i n

Vote

Global Icglobal and weights ( p1 ,..., pi ,..., pn )

Personal Ick and weights ( p ,..., p ,..., p ) 1 i n

Figure 4: An elementary vote stage with the (m+1) consistency indicators and the 2.m CITs. C. Basic examples of a vote session In a previous paper, Limayem and the author [13] presented a sophisticated two-stage vote session with two DMs. The DMs are experts in ergonomy. They confronted their opinions so as to result in the weighting of normalized ergonomic criteria for the optimization of a computer keyboard layout. We are giving here two basic examples before evoking the vote protocols we intend to implement on the web in the next chapter. In the simple example of figure 2, the method gives a consistency value of 99,74%, highlighting a very weak

inconsistency1. The CIT was in a major priority to increase the value of comparison c23. An increase of c23 from 0.5 to 0.6 allows to reach a 100% consistency. Finally, an example of an investment distribution between 4 actions (context not detailed here) was carried out in a marketing department. Here the group members directly converged towards a unique opinion for each comparison. They used different probability density functions (see the half comparison matrix in figure 5 because of reciprocity) in order to express their preferences at best. Action 2

Action 3

would be concerned. Indeed, for such companies, some important decisions based on the weighting of a set of elements oblige some executives to devote a whole day to travelling to participate to a physical meeting. Our method presents a number of degrees of freedom which are well suited to a remote web-based weighting session. The general framework would be the following: -

Action 4

Action 1 Action 2

Action 3 Figure 5: Comparison matrix for an investment distribution between 4 actions. For this software interface, a comparison c ij is not an assessment of the ratio p i / p j but is the percentage of the relative importance of element i versus the total (i+j), that is: 100 ⋅ p i / p i + p j .

(

)

The initial global consistency was 96.81% which already demonstrates a satisfactory quality of the decision. Increasing by 10% the relative importance c23 leads to a new global consistency of 98.82% and to the final weight vector in figure 6. The result clearly shows that there is no indifference between the elements (no pdfs overlappings). Action 1

Action 3

Action 2

Action 4

Figure 6: Final result of the weight vector after a two-stages session. D. A research in vote protocols for a web-based collaborative tool We gathered several testimonies from project leaders of large companies as far as using our method in an intranet context 1

Based on our experience, we estimated that the decision quality for one DM was good for a consistency above 95% and unacceptable below 80%.

The leader of the weighting session declares open a weighting session on a web server. He defines the theme of the vote, the number and the nature of the elements, the participants (names, skills, e-mail addresses), an imposed number or not of vote stages, a final date for the final result, a protocol vote (secret vote or not), a restriction of the DMs opinions to certain matrix lines or columns (depending on their skills)… The participants are warned that they are involved in a weighting session and they are free to fill their own comparison matrix Mk until the next result computation. Their personal consistency and the CIT relative to their own consistency improvement are computable at any moment. The web server automatically proceeds to the global weight vector, the global consistency indicator and the CITs relative to the improvement of the global consistency. The results are sent to the participants for a next stage or as end results.

For a number of vote sessions for which the stakes are not clear for the participants, it would be better to use webconferencing to discuss before voting individually. A mix of synchronous and asynchronous protocols, as well as secret and non secret vote protocols have to be defined. This is our current research activity.

IV. CONCLUSION After presenting the importance and the difficulties of weighting in a project, we have argued that Pairwise Comparison methods are recognized as convenient methods for weighting or ordering a set of elements in a group, even in the presence of different systems of preferences. Moreover, PC methods improve the rationality of the decision makers (DMs) and the rationality of the group because they are directly linked to the notions of ordinal transitivity and cardinal transitivity. For the purpose of understanding, Chapter 2 presents a famous PC method: the Least Square Logarithmic Regression and its extensions to uncertainty with fuzzy logic and probabilistic approach. Limayem [11] and the author [12; 13] have proposed some improvements for the two types of uncertainty extensions. In the context of our Monte Carlo Pairwise Comparison (MCPC) approach, we have developed some consistency indicators for measuring the degree of violation of the cardinal transitivity, generally considered with regards to the quality of the decision. These global (for the group) and personal (for each DM) consistency indicators are

accompanied by a twofold series of Consistency Improvement Tracks (CITs) letting free each DM to revise his/her judgment after personal or group considerations. Chapter 3 is completed by some basic examples of an elementary comparison matrix revision. Finally, a proposal of a framework for a web-based collaborative weighting method is presented. Different alternatives of vote protocols are currently under investigation. Such a functionality could allow to save time and money for certain types of projects and to popularise a valuable method to gain quality and to keep a record of the decision process. V. REFERENCES [1] Arrow K.J., (1951), Social Choice and Individual Values, NewYork, John Wiley. [2] Boender C.G.E., de Graan J.G., Lootsma F.A., (1989), Multicriteria decision analysis with fuzzy pairwise comparisons. Fuzzy Sets and Systems, vol. 29: p. 133 - 143. [3] Buckley J.J., (1985), Fuzzy hierarchical analysis. Fuzzy sets and systems, vol. 17: p. 233-247. [4] De Graan J.G., (1980), Extensions to the multiple criteria analysis of T. L. Saaty, Report National Institute of Water Supply, [5] Dong W.M., Wong F.S., (1987), Fuzzy weighted averages and implementation of the extension principle. Fuzzy Sets and Systems, vol.: p. 183-199. [6] Finan J.S., Hurley W.J., (1997), The analytical hierarchy process: does adjusting a pairwise comparison matrix to improve the consistency ratio help? Computers Operations Research, vol. 24(8): p. 749-755. [7] Hazelrigg G.A., (1996), The Implications of Arrow's Impossibility Theorem Approaches to Optimal Engineering Design. Journal of Mechanical Design, vol. 118: p. 161-164. [8] Hazelrigg G.A., (1997), On Irrationality in Engineering Design. Journal of Mechanical Design, vol. 119. [9] Herman M.W., Koczkodaj W.W., (1996), A Monte Carlo study of pairwise comparison. Information Processing Letters, vol. 57: p. 25-29. [10] Levary R.R., Wan K., (1998), A simulation approach for handling uncertainty in the analytic hierarchy process. European Journal of Operational Research, vol. 106: p. 116-122. [11] Limayem F., (2001), Modèles de pondération par les méthodes de tri croisé pour l'aide à la décision collaborative en projet. Thèse de doctorat soutenue le 23 novembre 2001, Ecole Centrale Paris. [12] Limayem F., Yannou B., (2000), A Monte Carlo Approach to Handle Imprecision in Pairwise comparison. in IDMME2000 : Third International Conference on Integrated Design and Manufacturing in Mechanical Engineering, Montréal. [13] Limayem F., Yannou B., (2002), Le Tri Croisé de Monté Carlo : une boite à outils pour l'aide à la décision coopérative. à paraître prochainement dans la Revue de CFAO et Informatique Graphique, vol. [14] Lootsma F.A., (1982), Performance evaluation of nonlinear optimization methods via multi-criteria decision analysis and via linear model analysis. M.J.D. Powell ed. Nonlinear Optimization 1981, ed. Press A. Vol. 1, London. 419-453. [15] Lootsma F.A., (2001), A revision of basic concepts in multicriteria decision analysis. [16] Saaty T.L., (1977), A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology, vol. 15(3): p. 234-281. [17] Saaty T.L., (1980), The Analytic Hierarchy Process, New-York, McGraw-Hill.

[18] Thurstone L.L., (1927), A law of comparative judgments. Psychological Reviews, vol. 34: p. 273-286. [19] Van Laarhoven P.J.M., Pedrycz W., (1983), A fuzzy extension of Saaty's priority theory. Fuzzy sets and systems, vol. 11: p. 229241.