Using Tocher's curve to convert subjective quantile ... - Springer Link

IIE Transactions (1999) 31, 245ÿ254

Using Tocher's curve to convert subjective quantile-estimates into a probability distribution function HON-SHIANG LAU1 , AMY HING-LING LAU1 and JOHN F. KOTTAS2;3 1

College of Business Administration, Oklahoma State University, Stillwater, OK 74078, USA Email: [email protected] 2 School of Business, College of William and Mary, Williamsburg, VA 23185, USA 3 Authors' names in reverse alphabetical order Received February 1997 and accepted August 1998

One standard approach for estimating a subjective distribution is to elicit subjective quantiles from a human expert. However, most decision-making models require a random variable's moments and/or distribution function instead of its quantiles. In the literature little attention has been given to the problem of converting a given set of subjective quantiles into moments and/or a distribution function. We show that this conversion problem is far from trivial, and that the most commonly used conversion procedure often produces large errors. An alternative procedure using ``Tocher's curve'' is proposed, and its performance is evaluated with a wide variety of test distributions. The method is shown to be more accurate than a commonly used procedure.

1. Introduction Many decision models contain distributions that have to be subjectively estimated, as is the case in PERT-simulation and Hertz-type [1] risk analysis. One standard way to estimate a subjective distribution is to elicit its quantiles from knowledgeable experts. However, implementing those decision models requires (typically) knowledge of the distributions' moments and/or distribution functions, but NOT of the quantiles. Very little research has been published in the literature on how the elicited quantiles can be converted into the moments and/or distribution functions required by the decision models. The implicit assumption is that the conversion problem is trivial. The objectives of this paper are to: (i) show that the conversion problem is far from trivial and ``solved''; the most frequently-used procedure and software often produce substantial errors; and (ii) evaluate in detail a much more accurate conversion procedure ÿ ®tting with a Tocher's curve. The remainder of this section further clari®es the relationships between ours and other similar problems in the literature. Sections 2 and 3 describe the two quantileto-distribution-function conversion procedures we are investigating: the Piecewise-Linear Approximation (PLA) and the Tocher-Curve (TC) procedures. Section 4 explains our methodology for evaluating these procedures' performance, and Section 5 reports our performance evaluation results. Section 6 addresses some likely questions about our evaluation methodology; a short concluding Section 7 follows. 0740-817X

Ó

1999 ``IIE''

1.1. De®nition of symbols For a random variable X , de®ne l ˆ E…X † to be its expected value; lk ˆ E…X ÿ l†k to be its kth central moment …k > 1†; a1 ˆ l3 =r3 to be its skewness measure (where p r ˆ l2 ); and b2 ˆ l4 =r4 to be its kurtosis measure. De®ne also b1 ˆ a21 . For our purpose, the term `®rst four moments' (€m) refers collectively to the l; r; b1 and b2 of the random variable. Note that an e€ective indicator of a distribution's shape is the pair (b1 ; b2 ). De®ne f …X †, F …X † and F ÿ1 … p† as, respectively, the true density function, the cumulative distribution function (cdf) and the inverse cdf of X . De®ne f^…X †, F^…X †, F^ÿ1 … p† as the ®tted density function, cdf and the inverse cdf produced by the conversion procedure. Let Xa denote the ath-quantile of X (where a, the quantile level, is a probability value); e.g., X0:5 is the median and X1 is the right-hand tail end. Note that for many theoretical distributions (e.g., normal), one or both tails are in®nite; i.e., X0 ˆ ÿ1 and/or X1 ˆ 1. When dealing with a series of quantiles (as in Section 2), X…i† denotes the ith quantile-value in the series (i here is an integer index). 1.2. Relationships among related problems in the literature For the purpose of this study, we identify two types of density-function estimation (or cdf estimation) problems: Type A. When a number (n) of actual (empirical) observations are available.

246

Lau et al.

Many methods have been developed for the solution of this type of problem. Standard textbook methods include the method of moments and the method of maximum likelihood. Other methods include those described in Silverman [2] and also the method of Avramidis and Wilson [3], in which a Johnson reference distribution is ®tted to the sample-€m and the n sample-quantiles are computed from the n empirical observations. Type B. When no empirical observation is available, and the random variable's statistical characteristics are subjectively estimated. This type can be further divided into Types B1 and B2: Type B1. Where there is no restriction on the type of statistical characteristics that can be subjectively estimated. Examples of estimated characteristics are: quantiles (including the endpoints X0 and X1 ), mode, l and/or r. The best-known example of a Type-B1 method is the PERT procedure [4]. The PERT formulas use subjective estimates of the mode (m), X0 and X1 to estimate a subjectivedistribution's mean (le ) and standard deviation (re ) via: le ˆ ……X0 ‡ 4m ‡ X1 †=6†, and re ˆ …X1 ÿ X0 †=6. Another method is via the use of a series of visual interactive software packages developed by DeBrota, et al. [5], AbouRizk et al. [6] and Wagner and Wilson [7]. Each of these packages allows a user to input an initial set of estimated characteristics (consisting of a mixture of quantiles and non-quantiles). The user then interacts with the software to produce a wide selection of ®tted distributions by revising the input set of estimated characteristics. Type B2. Where only subjective quantiles are available. There exists a large literature on subjective-quantile estimation (see, e.g., reference lists in such review articles as Hogarth [8] and van Lenthe [9]). Most of these papers assume explicitly or implicitly that one should elicit from human subjects only quantiles, but NOT: (i) X0 and X1 ; and (ii) the mode, l and/or r. For example, Alpert and Rai€a [10] have estimated subjective distributions by eliciting the following ®ve quantiles: X0:01 , X0:25 , X0:50 , X0:75 , and X0:99 . This quantile-only problem structure is the one addressed in most standard textbooks on decision making under uncertainty [11,12]. It is also widely adopted in recent applications of decision analysis [13,14]. This paper deals only with the Type-B2 situation: i.e., given a set of subjective quantiles, estimate the cdf and €m. Standard decision-analysis textbooks [15] prescribe the simple procedure of forming a piecewise-linear cdf by joining each adjacent pair of plotted quantiles, as shown in Fig. 1, with a straight line. A variant of this procedure is used in BESTFIT [16], which is a component in the increasingly popular risk-analysis software @RISK.

Fig. 1. A plot of quantiles.

Others have suggested that a cdf be obtained as ``a smooth curve drawn by eye through the plotted (quantile) points'' [17], this procedure is assumed to be so straightforward that no one seems to have studied: (i) exactly how one computerizes the drawing of the smooth curve; or (ii) the accuracy of the procedure. These questions are addressed in our study.

2. The piecewise-linear approximation procedure To convert a given set of quantiles to a continuous disÿ1 tribution function, the formation of F^ … p† by simply joining each adjacent pair of plotted quantiles with a straight line is a common procedure described in simulation textbooks [18ÿ20]. This is also a major procedure used by GPSS [21] to generate non-uniform random variates. 2.1. Formulae for computing the €m of the piecewise-linear approximation CDF Given any set of quantiles, the €m of the piecewise-linear ÿ1 F^ … p† can be computed with the formulae derived below. Let l0k ˆ E…X k † be the kth moment of the random variable X (about zero). It is known that [22]: l0k

Z1 ˆ

‰F ÿ1 … p†Šk dp:

…1†

0

(l0k )s,

Given the formulae [23]:

the (lk )s can be computed via standard

l2 ˆ l02 ÿ …l01 †2 ;

l3 ˆ l03 ÿ 3l01 l02 ‡ 2…l01 †3 ;

l4 ˆ l04 ÿ 4l01 l03 ‡ 6…l01 †2 l02 ÿ 3…l01 †4 :

…2†

Converting quantiles into probability

247

Thus, the remaining problem here is to compute l0k using (1). Now, Fig. 1 shows a plot of quantiles X…k† …k ˆ 1 to n) and their corresponding quantile levels (Pk )s. Each linear segment i (between Piÿ1 and Pi ) of the piecewise-linear function F^ÿ1 … p† can be expressed as X…i† ÿ X…iÿ1†  …p ÿ Piÿ1 †; Pi ÿ Piÿ1 ˆ X…iÿ1† ‡ Si …p ÿ Piÿ1 †; Piÿ1  p  Pi ;

F^ÿ1 … p† ˆ X ˆ X…iÿ1† ‡

…3†

where Si ˆ ‰X…i† ÿ X…iÿ1† Š=‰Pi ÿ Piÿ1 Š is the slope of the ith segment of F^ÿ1 … p†. Therefore, ZPi

^ÿ1

‰F

ZPi

k

… p†Š dp ˆ

Piÿ1

‰X…iÿ1† ‡ Si … p ÿ Piÿ1 †Šk dp;

(or (b1 ; b2 )s). A minor disadvantage is that, for some parameter-combination values, Fsÿ1 … p† may not be monotonically decreasing when p is close to 0 or 1. However, this tail-end irregularity should be judged in the perspective of the fact that a normal distribution with a small negative tail is routinely used to model a large variety of necessarily positive random variables. 3.2. Numerical illustration Assume the availability of the quantiles presented in (15b) and (15c) (see Section 4.2 below), where X0:05 ˆ ÿ0:198. Substituting this into (7) gives a ‡ 0:05b ‡ 0:0025c ÿ 2:703 65a ÿ 0:000 13b ˆ ÿ0:198: …8†

Piÿ1

h

i. k‡1 k‡1 ˆ X…i† ÿ X…iÿ1† ‰Si …k ‡ 1†Š:

…4†

Combining Equations (1) and (4) gives 9 8 P n‡1 < Z i = X ‰F ÿ1 … p†Šk dp ; l0k ˆ ; : iˆ1

2

Piÿ1

 n‡1 h i X k‡1 k‡1 ˆ X…i† ÿ X…iÿ1† ‰Si …k ‡ 1†Š :

…5†

iˆ1

Thus, substituting the quantile levels and values given in (15b) to (15d) (see Section 4.2) into (5) and (2) gives a f^…X † with ®tted moments: lf ˆ 1:481;

rf ˆ 3:051;

b1f ˆ 22:29;

Similarly substituting the other quantiles given in (15b) and (15c) leads to the linear-regression system TM ˆ Xa (bold letters indicate vectors/matrices), where T ˆ ‰a b c a bŠ (TC's parameters), Xa ˆ ‰ÿ0:198; ÿ0:066; 0:252; 0:783; 1:515; 2:355; 2:941Š, and M is the matrix

b2f ˆ 26:98: …6†

3. Tocher's curve

1

1

1

1

1

1

1

7 6 0:10000 0:25000 0:50000 0:75000 0:90000 0:95000 7 6 0:05000 7 6 7: 6 0:00250 0:01000 0:06250 0:25000 0:56250 0:81000 0:90250 7 6 7 6 4 ÿ2:70365 ÿ1:86509 ÿ0:77979 ÿ0:17329 ÿ0:01798 ÿ0:00105 ÿ0:00013 5 ÿ0:00013 ÿ0:00105 ÿ0:01798 ÿ0:17329 ÿ0:77979 ÿ1:86509 ÿ2:70365

Solving the above system gives ‰a b c a bŠ ˆ ‰ÿ0:2201; 2:0919; ÿ0:5814; 0:0291; ÿ0:6271Š: …9† ÿ1

The F … p† required in (1) is now fully de®ned by (7) and (9), and numerically integrating (1) using IMSL's [25] integration subroutine QDAGS gives: lf ˆ 0:997; rf ˆ 0:992; b1f ˆ 1:48; b2f ˆ 4:92

The poor performance of the piecewise linear approximation (PLA) method, that will be discussed in Section 5 motivated us to investigate various methods of `®tting a smooth curve' to the quantiles. For brevity's sake, we will only explain and report the results of the curve-®tting method that performed best: the Tocher-Curve (TC) method [24]. 3.1. Explanation of the TC method The Tocher inverse cdf can be expressed by Fsÿ1 … p† ˆ X ˆ a ‡ bp ‡ cp2 ‡ a…1 ÿ p†2 ln… p† ‡ bp2 ln…1 ÿ p†;

…7†

where a; b; c; a and b are parameters. The major advantages of Fsÿ1 … p† are: (i) it is a linear function of its parameters and can therefore be ®tted very easily to a given set of (®ve or more) quantiles by linear regression; and (ii) it has ®ve parameters and therefore has the potential of being able to model a large variety of distribution shapes

3

4. Approach for evaluating conversion procedures

…10†

quantile-to-distribution

We use a large number of known test distributions to evaluate the performance of the PLA and TC methods. For each test distribution, its quantiles are determined; these quantiles then become input data to the conversion procedure to be tested. One then evaluates how close the conversion-procedure-®tted distribution is to the known test-distribution. Explained below are the three basic components of this approach: (i) selection of test distributions; (ii) selection of quantiles to be considered; and (iii) criterion for evaluating the closeness between a test distribution and its ®tted counterpart. 4.1. Selection of test distributions We will evaluate PLA's and TC's performance with two groups of test distributions: ``Group A'' consists of 2000 beta distributions with di€erent randomly generated

248

Lau et al.

shapes, or (b1 ; b2 )s. ``Group B'' consists of 3000 distributions of di€erent (b1 ; b2 )s, with 1000 each from the Pearson, Johnson and RambergÿSchmeiser systems (these systems encompass a much larger variety of distribution shapes than the beta system). Distribution-theory related justi®cations for using these test distributions are detailed in Appendix A, which contains a brief review of the (b1 ; b2 ) diagram. Brie¯y illustrated below is the generation of one test distribution in Group A. The beta distribution's density function can be expressed as …X ÿ U †pÿ1 …V ÿ X †qÿ1 ; …11† f …X † ˆ B…p; q†…V ÿ U †p‡q‡1 where B…:; :† is the beta function, …U ; V † are the end points (X0 ; X1 ), and …p; q† are the shape parameters. To simplify the evaluation of the conversion procedure's performance (clari®ed in Section 4.3), we will use standardized beta distributions with l ˆ r ˆ 1. Thus, the ®rst of 2000 randomly generated shapes is (b1 ; b2 † ˆ …1:56; 5:095); hence the true €m of distribution #1 are: l ˆ 1;

b1 ˆ 1:560;

r ˆ 1;

b2 ˆ 5:095:

…12†

Substituting these €m into standard closed-form formulas [26] gives the beta-distribution's parameters (see Equation. 11): …U ; V ; p; q† ˆ …ÿ0:451; 23:085; 1:913; 29:127†:

…13†

4.2. Selection of quantiles to be considered An increasingly large number of theoretical, empirical and case studies [10,27ÿ29] support the notion that a subjective distribution should be constructed by eliciting its median (i.e., X0:5 ) plus four or six symmetrical quantiles from the following set of quantile-levels (ai )s (i 2 S): Index setS ˆfi;iˆ1;9g Quantiles levelai

1

2

3

4

5

6

7

8

9

0:01 0:05 0:10 0:25 0:50 0:75 0:90 0:95 0:99

…14† We consider in this study all the following ten possible di€erent subsets of S with four or six symmetric (ai )s (plus ai ˆ 0:5): S1 ˆ f1 2 3 5 7 8 9g; S2 ˆ f1 2 4 5 6 8 9g; S3 ˆ f1 3 4 5 6 7 9g; S4 ˆ f2 3 4 5 6 7 8g; S5 ˆ f1 2 5 8 9g; S6 ˆ f1 3 5 7 9g; S7 ˆ f1 4 5 6 9g; S8 ˆ f2 3 5 7 8g; S9 ˆ f2 4 5 6 8g; S10 ˆ f3 4 5 6 7g:

To illustrate numerically: when S4 is used, the corresponding quantiles for the beta distribution de®ned by Equations (12) or (13) are given in (15c): Set S4 ; with i ˆ

2

3

4

5

6

7

8

…15a†

Quantile level ai

0:05

0:10

0:25

0:50

0:75

0:90

0:95

…15b†

ÿ0:198 ÿ0:066 0:252 0:783 1:515 2:355 2:941

…15c†

Quantile value X…i† ˆ Xai

Incidentally, (13) also shows that the beta-distribution de®ned by (12) has: X0 ˆ ÿ0:451;

X1 ˆ 23:085:

…15d†

Procedures for computing quantiles such as those in (15) for our test distributions are outlined in Appendix B. 4.3. Evaluating the closeness between a test and its ®tted distributions The quantiles in (15aÿd) for a given test distribution f …X † are used as the inputs to a conversion procedure; the outputs are the f^…X † and its moments, as illustrated in (6) and (10). This study evaluates the closeness between f …X † and f^…X † by considering the di€erences of their €m. Thus, comparing (6) with (12), the closeness measures (or moment deviations) are expressed as: 9 Dl ˆ jlf ÿ lj ˆ 0:481; or 48:1% of r; > > Dr ˆ jrf ÿ rj ˆ 2:051 or 205:1% of r; = …16† Db1 ˆ jb1f ÿ b1 j ˆ 20:73; > > ; Db2 ˆ jb2f ÿ b2 j ˆ 21:89: If X is not standardized with l ˆ r ˆ 1, then the deviation in rf should be evaluated in terms of [(rf ÿ r†=rŠ. Also, lf should then be evaluated in terms of [(lf ÿ l†=rŠ; note that the obvious alternatives of considering either D1 ˆ …lf ÿ l† or D2 ˆ ‰…lf ÿ l†=lŠ are inappropriate because both D1 and D2 can be made arbitrarily small by using a small r, and D2 can also be made arbitrarily small by using a large l. Since our test distributions have small ranges of b1 and b2 (see Appendix A), deviations in b1f and b2f can be expressed in terms of (bif ÿ bi ) regardless of whether X is standardized with respect to l and r. The literature on decision models suggests that the €m-deviations presented in (16) are among the most important criteria for evaluating the closeness between f …X † and f^…X †. Other closeness criteria will be discussed in Section 6.4. 4.4. An important di€erence between PLA's and TC's quantile requirement A closer examination of the derivation of (5) reveals that, under PLA, a F^ÿ1 … p† can only be constructed from a set of quantiles that includes ®nite X0 and X1 . However, for most common distributions other than the beta distribution, at least one of X0 and X1 is in®nite (see Appendix A and Fig. A1 for further explanation). We therefore established the Group A test distributions in order to

Converting quantiles into probability

249

evaluate this well-known conversion procedure. However, even for bell-shaped beta distributions that have a ®nite X0 and X1 , the extensive quantile-elicitation literature suggests that subjective estimates of X0 and X1 should not be elicited because they tend to have much larger estimation errors than other quantiles. Nevertheless, we will assume that X0 and X1 are given (only) under this conversion procedure. That is, in using any given quantile set Sk (e.g., S4 in (6a)), the PLA procedure will actually be given two more (error-free) quantiles than the alternative TC method (which requires neither X0 nor X1 ). Since (shown in Section 5) the PLA method turns out to be so inaccurate even under all these accommodations, the issue of unfair advantage becomes irrelevant.

5. Performance results for the PLA and TC methods 5.1. Evaluating the piecewise-linear approximation procedure The procedure illustrated in equations (6) and (16) was applied to 999 additional Group A test distributions, and the average moment deviations were calculated. The procedure was then repeated for a second subsample of 1000 test distributions to ascertain that the average moment deviations obtained from the two subsamples are close. The average moment deviations for the aggregate sample (Group A's 2000 distributions) was then computed for Table 1 (this approach will also be used in all later evaluations to con®rm that a suciently large sample size has been used in each case). The above procedure was performed for all 10 (Sk )s enumerated in Section 4.2. The aggregate results lead to the same conclusions apparent from the results for S4 and S3 reported in Table 1.

Since the moment deviations for PLA are large, we veri®ed the correctness of (5) by: (i) simulation (i.e., generating random variates from F^ÿ1 … p† and computing the variates' sample €m); and (ii) comparing our computed answers with the answers produced by the BESTFIT software (see Section 5.2). The primary reason for the large errors observed for the PLA method is that the linearized tails of the piecewiselinear f^…X † are usually much thicker than the f …X †'s real tails, and these thicker tails result in moments of in¯ated magnitudes. We have found that the moment deviations can be substantially reduced by providing intentionally erroneous X0 and X1 that constitute a narrower range than the true X0 and X1 . However, the moment deviations after these improvements are still much larger than those of the other conversion procedures we considered, therefore these improvement e€orts are not reported. Recall that the PLA method cannot handle a Group B test distribution, where at least one of X0 and X1 is in®nite. 5.2. The BESTFIT procedure (a PLA related procedure) 5.2.1. For lf and rf Palisade Corporation informed us that BESTFIT obtains lf and rf using the PLA method, and we veri®ed this by comparing the (lf )s and (rf )s obtained from running the BESTFIT software with those computed by our PLA computer program. Therefore BESTFIT's average Dl and Dr are the same as those of the PLA method (see Table 1). 5.2.2. For b1f and b2f In contrast, BESTFIT obtains its initial estimates for b1f and b2f via a linear-piecewise-related procedure, but these initial estimates are improved through a numerical search

Table 1. Comparative average moment deviations from the piecewise-linear, BESTFIT and Tocher-curve methods. Two testdistribution groups (A and B), and two quantiles sets: S4 = {2, 3, 4, 5, 6, 7, 8} and S3 = {1, 3, 4, 5, 6, 7, 9} Quantile set

Performance criterion

Group A Test distributions PLA method

BESTFIT software

TC method

Group B test distributions TC method

S4

Dl (% of r) Dr (% of r) Db1 Db2

162 793 13.7 14.7

162 793 1.16a 9.14a

0.18 0.82 0.120 0.310

0.35 3.30 0.370 1.86

S3

Dl (% of r) Dr (% of r) Db1 Db2

36 342 42.1 51.6

36 342 note b note b

0.04 0.13 0.048 0.163

0.04 0.52 0.193 1.25

0.0002+?c

0.02

0.02

CPU seconds (with an IBM 3090-200s):

0.0002

Note: a20 test problems for BESTFIT-®tted b1f and b2f b Did not investigate c BESTFIT ®rst executes the PLA method, then improves the PLA-generated b1f and b2f with a proprietary numerical search procedure. Hence the required CPU time must exceed PLA's 0.0002 seconds

250 procedure to obtain the ®nal b1f and b2f . However, we are unable to obtain sucient details from the Palisade Corporation about the procedure to program it. Further, it is impractical to use the BESTFIT software to process 2000 test problems since the software requires each problem to be keyed-in and solved individually. We therefore ran the BESTFIT software for the ®rst 20 of our 2000 randomlygenerated distributions. The resulting average Db1 and Db2 are listed in Table 1. Although they are smaller than PLA's average Db1 and Db2 , they are still substantially larger than the average Db1 and Db2 of the other methods (of which only the TC method is reported here) we tested. Adding this to their large Dl and Dr (which are based on 2000 test distributions), we feel that it is not worthwhile to use more test distributions to further evaluate BESTFIT's performance for our problem. 5.3. Evaluating the Tocher curve procedure The TC procedure was evaluated with the same quantilesets (Sk )s (k ˆ 1±10) from the Group A and B test distributions used earlier to evaluate the PLA and BESTFIT procedures. For each quantile-set Sk of each test distribution, the required evaluation computations were illustrated earlier in Section 3.2 and (16). In Table 1, not only are the Group A distributions' average moment deviations one to several orders of magnitude smaller in the TC than in the PLA method, but these substantial performance di€erences are largely retained even when one compares the PLA method under Group A test distributions with the TC method under Group B distributions. Compare now the Group A and Group B average moment deviations of the TC method. As expected, the average moment deviations deteriorate when Group B distributions are used instead of Group A. However, the deterioration is surprisingly small for the average Dl and quite tolerable for the average Dr and Db1 . The deterioration is substantial only for the average Db2 . This latter deterioration is due to two factors: (i) we found that a test distribution's Db2 is positively correlated with its true b2 ; (ii) as explained in Appendix A, Group B contains a very much higher proportion of high b2 distributions than Group A. Nevertheless, the inaccuracy of a TC-estimated b2f (which is after all much less serious than a PLA-estimated b2f ) should be viewed from the following perspective: De®ne x, s, b1 and b2 as, respectively, the sample estimates of l, r, b1 and b2 . If one is using empirical sample observations to estimate a random variable's €m, it is known that (as discussed by Kendall et al. [23]) the sampling error of the sample estimates increases rapidly as one moves from x to s to b1 and to b2 . The sampling error for b2 also increases rapidly with b2 . In other words, the real b2 of a high b2 distribution is dicult to estimate reliably even with a fairly large sample (say, sample size ˆ 1000).

Lau et al. Although the TC method's superiority has been proven so far only with respect to the ten Sk quantile-sets, we have con®rmed its expected superiority to the PLA method by repeating our performance evaluation but using instead quantiles at randomly-generated quantilelevels (results are not reported). Given the superior accuracy attainable with the Tocher's curve, one should then note that its inverse cdf form allows the use of the inverse-transformation method of variate generation and hence greatly simpli®es risk simulation ÿ an important solution methodology in risk analysis involving subjective random variables. The relatively high CPU time (0.02 seconds) of the TC method (see Table 1) is due to the €m-computing numerical integration procedure required by (1). Recall that, in contrast, for the PLA method the required integrations pertaining to (1) could be done with the closed-form formula (5). Nevertheless, our guess is that the TC procedure's CPU time should still be less than that required by BESTFIT's proprietary numerical optimization procedure (for improving its b1f and b2f ). In any case, the 0.02 second of mainframe CPU time still translates to a practically instantaneous answer from a personal computer (some of which may be faster than a mainframe).

6. Justi®cations of certain approaches used in this study This section considers two aspects of our study that are most likely to be questioned. Appendix A handles a third aspect that relates to distribution theory: did we consider test distributions of all possible forms? 6.1. Necessity to test for non-beta parent distributions One might argue that the Group B distributions are irrelevant because the beta distribution can model most real-life subjective (as opposed to empirical) distributions. Despite the huge literature on subjective quantile elicitation methodologies, the only elicited quantiles we are able to ®nd in the open literature are the ones in Solomon [28] and Shields et al. [13]. For example, Shields et al. [13] elicited from accountants their subjective quantiles on the audited value of the ``Sales'' and other accounts of a ®rm. Fitting their reported quantiles with the Tocher's curve gives the ®tted b1 and b2 values reported in Table 2. Table 2 shows that all six subjective distributions have non-beta (b1 ,b2 )s (i.e., as clari®ed in Appendix A, these (b1 ,b2 )s all lie below the beta-distribution area in Fig. A1). Thus, there is no evidence that subjective distributions are less likely to be non-beta than empirical distributions. 6.2. Other criteria for measuring closeness between f…X† ^ and f…X† Consider the following two additional closeness criteria: for each test distribution, de®ne fx…i†; i ˆ 1; 51g as 51

Converting quantiles into probability

251

Table 2. (b1 ; b2 )s of real-life subjective distributions Account

Sales

Cost of sales

Net inventory

Net accounts receivable

Accounts payable

Depreciation expenses

Fitted b1 Fitted b2

0.03 4.57

0.15 4.68

0.01 4.33

0.01 3.39

1.79 7.09

0.12 4.14

Table 3. Average DK and DA for the piecewise-linear and Tocher-curve methods Group A distributions, (Sk)s are: S4 = {2, 3, 4, 5, 6, 7, 8} and S3 = {1, 3, 4, 5, 6, 7, 9} Quantile set

Performance criterion

Fitting method PLA

S4 = {2, 3, 4, 5, 6, 7, 8}

DK DA DK DA

S3 = {1, 3, 4, 5, 6, 7, 9}

equally-spaced x values between X0:01 and X0:99 . We then compute values of F ‰x…i†Š and F^‰x…i†Š for each i. The ®rst additional closeness criteria is a KolmogorovÿSmirnov type criterion DK ˆ maxjF ‰x…i†Š ÿ F^‰x…i†Šj: i

The second is an average of the cdf deviations over the range; i.e., ( ) X 51: jF ‰x…i†Š ÿ F^‰x…i†Šj DA ˆ i

Table 3 gives the average DK and DA values over the 2000 Group A test distributions for the PLA and TC methods. Although Table 3 con®rms the superiority of the TC method, its more useful purpose here is to demonstrate the importance of using ``relevant'' and suciently discriminating performance criteria. Firstly, in most decision scenarios, the mean and variance of a decision criterion (e.g., a project's NPV) are more relevant than the decision-criterion's (e.g, NPV's) quantile-value at some quantile level. In other words, the moment deviations considered in Table 1 are much more relevant than the criteria DK and DA considered here. Secondly, Table 3 shows that for the criterion DK, PLA does not perform much worse than TC. In other words, had DK been used as the performance criterion, it would have concealed PLA's unacceptable performance with respect to the much more relevant criteria Dl and Dr. Actually, a moment's re¯ection will reveal the inherent weakness of DK-type criteria: they are based on di€erences in cumulative probabilities, which have very small ranges. Therefore DK-type values will always be ``respectably small'' regardless of how poor the ®t is. A worse variant of the DK=DA-type criteria is one that considers cdf-deviations jF …X † ÿ F^…X †j only at the subjectively-estimated quantile-levels ai 's. Since PLA forces the ®tted cdf to pass through (exactly) each given quantile, this

0.327 0.103 0.206 0.939

E)1 E)1 E)1 E)2

TC 0.207 0.270 0.163 0.322

E)1 E)2 E)1 E)1

latter DK=DA variant would indicate that PLA performs perfectly (i.e., DK ˆ DA ˆ 0)!

7. Conclusion We have shown that, for ®tting a set of quantiles to their parent distribution, the well-known PLA method and its BESTFIT variant often produce substantial errors. We then showed that a ®tting procedure based on the Tocher's curve is much more accurate.

References [1] Hertz, D. (1964) Risk analysis in capital investment. Harvard Business Review, 42, 95ÿ106. [2] Silverman, B. (1986) Density Estimation for Statistics and Data Analysis, Chapman and Hall, London. [3] Avramidis, A. and Wilson, J. (1994) A ¯exible method for estimating inverse distribution functions in simulation experiments. ORSA Journal on Computing, 6(4), 342ÿ355. [4] Malcolm, D., Roseboom, J., Clark, C. and Fazar, W. (1959) Application of a technique for research and development program evaluation. Operations Research, 7, 646ÿ669. [5] DeBrota, D., Dittus, R., Roberts, S. and Wilson, J. (1989) Visual interactive ®tting of bounded Johnson distributions. Simulation, 52, 199ÿ205. [6] AbouRizk, S., Halpin, D. and Wilson, J. (1991) Visual interactive ®tting of beta distributions. Journal of Construction Engineering and Management, 117(4), 589ÿ605. [7] Wagner, M. and Wilson, J. (1996) Using univariate Bezier distributions to model simulation input processes. IIE Transactions, 28, 699ÿ711. [8] Hogarth, R. (1975) Cognitive processes and the assessment of subjective probability distributions. Journal of the American Statistical Association, 70, 271ÿ294. [9] van Lenthe, J. (1994) Scoring-rule feedforward and the elicitation of subjective probability distributions. Organizational Behavior and Human Decision Processes, 59, 188ÿ209. [10] Alpert, M. and Rai€a, H. (1982) A progress report on the training of probability assessors, in Judgement under Uncertainty: Heuris-

252

[11] [12] [13]

[14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37]

Lau et al. tics and Biases, Kahneman, D., Slovic, P. and Tversky, A. (eds.), Cambridge University Press, New York, NY. Holloway, C. (1979) Decision Making under Uncertainty, PrenticeHall, Englewood Cli€s, NJ. Samson, D. (1988) Managerial Decision Analysis, Irwin, IL. Shields, M., Solomon, I. and Waller, W. (1987) E€ects of alternative sample space representations on the accuracy of auditors' uncertainty judgments. Accounting, Organization and Society, 12(4), 375ÿ385. Keefer, D. and Verdini, W. (1993) Better estimation of PERT activity time parameters. Management Science, 39(9), 1086ÿ1091. Clemen, R. (1991) Making Hard Decisions, Duxbury Press, Belmont, CA. Palisade Corporation, (1996) BESTFIT User's Guide, @RISK User's Guide, New®eld, NY. Hertz, D. and Thomas, H. (1983) Risk Analysis and Its Applications, John Wiley, New York, NY. p. 161. Law, A. and Kelton, W. (1982) Simulation Modeling and Analysis, McGraw-Hill, New York, NY. Thesen, A. and Travis, L. (1992) Simulation for Decision Making, West Publishing, St. Paul, MN. Banks, J., Carson, J. and Nelson, B. (1996) Discrete-Event System Simulation, Prentice-Hall, Englewood Cli€s, NJ. Schriber, T. (1974) Simulation Using GPSS, John Wiley, New York, NY. Ramberg, J. and Schmeiser, B. (1974) An Approximate method for generating asymmetric random variables. Communications of the ACM, 17(2), 78ÿ82. Kendall, M., Stuart, A. and Ord, J. (1987) Kendall's Advanced Theory of Statistics, Vol. 1, Oxford University Press, New York, NY. ch 10. Tocher, K. (1963) The Art of Simulation, English Universities Press, London. Anon, (1994) IMSL (1994), in IMSL Math/Library and IMSL Stat/Library, Visual Numerics Inc., Houston, TX. Johnson, N., Kotz, S. and Balakrishnan, N. (1995) Continuous Univariate Distributions, Vol. 2, John Wiley, New York, NY. Selvidge, J. (1980) Assessing the extremes of probability distributions by the quantile method. Decision Sciences, 11, 493ÿ502. Solomon, I. (1982) Probability assessment by individual auditors and audit teams: an empirical investigation. Journal of Accounting Research, 20, 689ÿ710. Lau, A., Lau, H. and Zhang, Y. (1996) A simple and logical alternative for making PERT time estimates. IIE Transactions, 28, 183ÿ192. Perry, C. and Greig, I. (1975) Estimating the mean and variance of subjective distributions in PERT and decision analysis. Management Science, 21, 1477ÿ1480. Hahn, G. and Shapiro, S. (1967) Statistical Models in Engineering, John Wiley, New York, NY. Fama, E. (1965) Portfolio analysis in a stable Paretian market. Management Science, 11, 404ÿ419. Bowman, K. and Shenton, L. (1979) Approximate percentage points for Pearson distributions. Biometrika, 66, 127ÿ132. Bowman, K. and Shenton, L. (1979) Further approximate Pearson percentage points and CornishÿFisher. Communications in Statistics, B8(3), 231ÿ244. Bowman, K. and Serbin, C. (1981) Explicit approximate solutions for SB . Communications in Statistics, B10(1), 1ÿ15. Bowman, K. and Shenton, L. (1980) Evaluation of the parameters of SU by rational fractions. Communications in Statistics, B9(2), 127ÿ132. Ramberg, J., Dudewicz, E., Tadikamalla, P. and Mykytka, E. (1979) A probability distribution and its uses in ®tting data. Technometrics, 21(2), 201ÿ214.

Appendix A Groups A and B test distributions: details and justi®cations Group A distributions In earlier e€orts to develop formulae to estimate a distribution's l and r from a given set of quantiles such as those of Malcolm et al. [4], Perry and Greig [30], Keefer and Verdini [14] and Lau et al. [29], it was assumed that the quantiles came from a bell-shaped beta parent distribution. Therefore it appears reasonable to expect a conversion procedure to perform well at least for our Group A distributions. Beta distributions also have ®nite tails (i.e., x0 and x1 ), making it feasible to evaluate the Piecewise-Linear-Approximation (PLA) method and its variant (the BESTFIT software), since PLA cannot handle distributions with in®nite tails. The 2000 di€erent (b1 ; b2 )s (for the 2000 test distributions) are randomly generated from the (b1 ; b2 ) area of the bell- (i.e., inverted-U) shaped beta (see the (b1 ; b2 ) diagram reproduced here as Fig. A1). Note that we only use distributions with di€erent (b1 ; b2 ) values and not with di€erent l and r values. This is because it can be shown that a conversion procedure's accuracy is a€ected by a distribution's shape but not by its location or scale. Similarly, a conversion-procedure's accuracy remains unchanged when a positively-skewed distribution is switched to its negatively-skewed mirror image. pHence, we will only consider distributions with a1 ˆ j b1 j. Incidentally, in case one questions the validity of excluding U- and J-shaped distributions, we have repeated our investigations by using the entire Type I area (results not reported) to con®rm that the conclusions reported in this paper are not a€ected by whether the U- and J-shaped are excluded. Group B distributions Although the bell-shaped beta distribution has been perceived to be suciently versatile in many papers that modeled real-life random variables, Figure A1 illustrates that it (i.e., the ``Type I'' Pearson distribution) covers only a very small portion of the entire feasible (b1 ; b2 ) area. While the (b1 ; b2 )-area above the bell-shaped-beta area consists of J- and U-shaped distributions that are generally recognized as somewhat irrelevant for subjective distributions, it is well-established that many real-life empirical distributions have (b1 ; b2 )s in the vast (b1 ; b2 )area below the bell-shaped-beta's area (see also Section 6.1). It takes all three ``main types'' of the Pearson system of distributions (namely, Types I, IV and VI) to cover the entire feasible (b1 ; b2 )-area. This point is explained in standard statistics textbooks such as Hahn and Shapiro [31]. However, one can further argue that a distribution having a certain (b1 ; b2 ) need not necessarily be a Pearson distribution. Two other distribution systems with wide

Converting quantiles into probability

253

Fig. A1. The b1 , b2 diagram, (adapted from Hahn and Shapiro [31]).

(b1 ; b2 ) coverage considered in the IE/MS/OR literature are the Johnson [31] and the RambergÿSchmeiser [22] systems. Hence the inclusion of these two systems with Pearson's system in Group B. We argued earlier with Fig. A1 that the (b1 ,b2 )-strip for the bell-shaped beta distributions is too small. On the other hand, using the entire possible (b1 ,b2 )-area with both b1 and b2 ranging to in®nity would be excessive because: (a) a conversion procedure that can do well in this in®nitely large (b1 ,b2 )-area probably does not exist; and (b) most realistic distributions do not have extremely high (b1 ,b2 )'s. We follow most standard textbooks/papers in distribution theory by considering the ``relevant'' (b1 ,b2 )-area bounded as: fb1  4; b2  10; …b1 ; b2 † below line BB in Fig. A1g: …A1† One thousand (b1 ,b2 ) values are randomly generated from this relevant (b1 ,b2 )-area, and each (b1 ,b2 ) de®nes three standardized (i.e., l ˆ r ˆ 1) distributions: one Pearson, one Johnson, and one RambergÿSchmeiser. This gives a total of 3000 test distributions in Group B. Failure to consider all possible parent distributions It is apparent that our test distributions do not include: (a) the entire feasible (b1 ,b2 )-area; and (b) all possible

distribution forms within the limited (b1 ,b2 )-area that Groups A and B cover. Regarding factor (a), the limited (b1 ,b2 )-area we considered was brie¯y justi®ed preceding its de®nition in (A1). We have repeated the experiments described in Section 5.3 with enlarged (b1 ,b2 )-areas (with b1 up to eight and b2 up to 20) to con®rm that the TC-procedure's performance is robust with respect to these changes. However, as explained below, the procedure may produce large moment-deviations if the parent distribution has extremely high (b1 ,b2 ) values. Regarding the factor (b), we recognize that, theoretically, a given set of quantiles (as in (15(aÿd))) can correspond to an in®nite number of parent distribution functions. Hence it is always possible to ``trick'' any given conversion procedure by devising a f …X † that will lead to large conversion errors. As an extreme example, assume that the (unknown) parent distribution is ``stable Paretian'', this distribution was widely advocated for modeling a large variety of ®nancial variables [32], and has r ˆ b1 ˆ b2 ˆ 1. Using a ``right'' combination of parameters, a stable-Paretian distribution can have ``true'' quantiles that are very similar to (say) those given in (15b) and (15c), and hence these quantiles would lead a conversion procedure to produce some ®nite rf , b1f and b2f , hence Dr ˆ Db1 ˆ Db2 ˆ 1! Thus, it is unrealistic to expect a conversion method to do well for all possible f …X † values. Our consideration of the (b1 ,b2 )-area de®ned in (A1) and the three distribution systems included in Group B represents a ``reasonable'' coverage of likely distributions typically assumed in the IE/MS/OR literature.

Appendix B Outline on generating percentiles for various distribution systems Handling Pearson distributions The method of Bowman and Shenton [33, 34] was used to generate the Pearson-distributions' quantiles. Bowman and Shenton have con®rmed this method's reliability only for distributions with b1  4. Handling Johnson distributions For a given (b1 ,b2 ), the Johnson-distribution's shape parameters c and g were determined by a FORTRAN program implementing the procedure described in: (a) Bowman and Serbin [35] for a SB distribution; and (b) Bowman and Shenton [36] for a SU distribution. Given the shape parameters c and g, a kth percentile (Jk ) of a Johnson distribution with location parameter  ˆ 0 and scale parameter k ˆ 1 can be obtained by transforming the corresponding standard normal percentile Nk as follows:

254

Lau et al.

for SB : y ˆ …Nk ÿ c†=g; then Jk ˆ 1=‰1 ‡ eÿy Š; for SU : y ˆ …Nk ÿ c†=g; then Jk ˆ sinh…y†:

…A2†

The mean (l0 ) and standard deviation (r0 ) of a Johnson distribution with  ˆ 0 and k ˆ 1 and density function fJ …
† can be determined by numerically integrating the expressions Z1 l0 ˆ

XfJ …X † dX ; ÿ1

r0 2 ˆ

Z1

2

…X ÿ l0 † fJ …X † dX : ÿ1

…A3† The quantile Jk0 for a distribution with l ˆ r ˆ 1 can then be computed from the corresponding Jk (see (A.2)) as Jk0 ˆ …Jk ÿ l0 †=r0 ‡ l:

…A4†

Handling RambergÿSchmeiser distributions For a randomly given (b1 ; b2 ), the values of the RambergÿSchmeiser distribution's shape parameters k3 and k4 were obtained by numerically solving the nonlinearprogramming problem explained in Ramberg et al. [37]. Initial values for the numerical optimization procedure were obtained from the tables of Ramberg et al. using linear interpolation. After obtaining k3 and k4 , the location and scale parameters k1 and k2 can be computed with closed-form formulae (given in Ramberg and Schmeiser [37]) in terms of k3 , k4 , and the desired l and r. Given the distribution's parameters, the kth quantile Rk (e.g., R0:95 ,

where k is expressed as a fraction) can be easily computed from the distribution's closed-form inverse cumulative distribution function: Rk ˆ k1 ‡ ‰k k3 ÿ …1 ÿ k†k4 Š=k2 :

…A5†

Biographies Hon-Shiang Lau is Regents Professor and Carson Professor of Management at Oklahoma State University. He graduated from Catholic High School in Singapore, and received his Ph.D. in Business Administration from the University of North Carolina at Chapel Hill. His research interests range from human resource accounting to production line design to commodity futures. The results reported in more than 100 refereed articles have appeared in, e.g., The Accounting Review, Decision Sciences, EJOR, IIE Transactions, IJPR, Journal of Business & Economic Statistics, Management Science, etc. Amy Hing Ling Lau is Regents Professor and Kerr-McGee Professor of Accounting at Oklahoma State University. She graduated from St. Nicholas High School in Singapore and received her MPA and Ph.D. from Texas Christian University and Washington University respectively. Among her publications are numerous articles in such journals as The Accounting Review, Contemporary Accounting Research, Decision Sciences, IIE Transactions, Journal of Accounting Research, Journal of Business and Economic Statistics, Management Science, etc. John F. Kottas is Zollinger Professor of Business at the College of William and Mary. He received his Ph.D. from Northwestern University. His research interests include risk analysis, scheduling, line design, and competitive bidding. He has coauthored four textbooks and numerous research papers.