Estimates of the Dollar Value of Employee Output in ...

Journal of Applied Psychology 1992. Vol. 77, No. 3, 234-250

Copyright 1992 by the American Psychological Association, Inc. 0021-9010/92/53.00

Estimates of the Dollar Value of Employee Output in Utility Analyses: An Empirical Test of Two Theories Michael K. Judiesch, Frank L. Schmidt, and Michael K. Mount Department of Management and Organizations University of Iowa This study examined distributions of estimates of the dollar value of performance in studies employing Schmidt, Hunter, McKenzie, and Muldrow's (1979) method for estimating the standard deviation of job performance (SDy) and found evidence that (a) the mean 50th percentile estimate is biased downward, (b) estimates of SDy appear to be a constant percentage of the 50th percentile estimate, and (c) estimates of SDy as a percentage of the 50th percentile value (SDP) are quite similar to empirical SDP values based on actual employee output. These findings suggest that the downward bias in the mean estimate of the 50th percentile causes the mean estimate of SDy to be similarly biased downward, but does not bias the estimates of SDP. Finally, an objective method for estimating the value of average employee output is described. We conclude that the product of this value and the mean supervisory estimate of SDP yields an unbiased estimate of SDy.

Much recent research in the area of selection utility has focused on the psychometric properties of supervisory estimates obtained with the SDy estimation procedure developed by Schmidt, Hunter, McKenzie, and Muldrow (1979). One psychometric issue that has received attention is that of interjudge agreement. Schmidt et al.'s procedure requires supervisors to estimate the current yearly dollar value of output as sold (i.e., gross sales revenue) for employees at the 15th, 50th, and 85th percentiles. The difference between the 50th and 15th percentile estimates and the difference between the 85th and 50th percentile estimates are computed for each judge. These differences are then averaged across judges to estimate SDy. Researchers have typically found a substantial amount of variability in the individual supervisory estimates of each percentile and of SDy(Eobko, Karren, & Parkington, 1983; Burke & Frederick, 1984; Greer & Cascio, 1987; Pearlman, 1985; Rich & Boudreau, 1987; Schmidt et al., 1979; Weekley, Frank, O'Connor, & Peters, 1985). However, as Schmidt et al. noted, unless there is an upward or downward bias in the group as a whole, averaging these estimates across a large number of judges will result in a fairly accurate mean estimate ofSDy. Nevertheless, the large variability in supervisory judgments is potentially problematic. As Boudreau (1991) noted, high interrater variability may indicate the presence of bias in the mean SDy estimate. Support for the hypothesis that the mean SDy estimate may be biased comes from the conclusion of several researchers that judges appear to use different interpretations of dollar value in making their dollar estimates (Bobko et al., 1983; Greer & Cascio, 1987; Reilly&Smither, 1985; Weekley et al., 1985). Bobko, Karren, and Kerkar (1987) argued that research aimed at understanding the cognitive processes underlying judgments of dollar value is needed to explain why this

possible variation in scale exists. More recently, Campbell (1990) called for research aimed at understanding the decisionmaking processes used by judges to scale their perceptions of performance against a dollar metric. The study reported here is based on the premise that an understanding of how judges go about making their percentile estimates can be gained by analyzing the psychometric properties of supervisory estimates. First, we present evidence that most judges have not interpreted Schmidt et al.'s (1979) estimation task as Schmidt et al. intended. Specifically, we present data that suggest that most judges appear to equate the 50th percentile value with average wages and that wages are much smaller than the revenue value of the output of the 50th percentile employee. This suggests that the majority of judges use a scale other than that of sales revenue and that the result is a downward bias in the mean 50th percentile estimate. However, our conclusion that the 50th percentile estimate is underestimated does not necessarily imply that SDy is underestimated. Whether underestimation of percentile values by supervisors results in a downward bias in the difference between percentiles depends on the relationship between the alternative scales that supervisors use and the revenue scale. Therefore, we developed and tested two competing theories about how judges make their estimates of the other two percentiles after making their 50th percentile estimate. The two theories derive from the fact that underestimation of the 50th percentile makes it impossible to accurately estimate simultaneously both the standard deviation in dollar terms (SDy) and the coefficient of variation of job performance (SDy/Y, or SDP) from supervisory estimates of the three percentiles. SDP is usually computed from studies that provide distributions (physical counts) of actual employee output (Hunter, Schmidt, & Judiesch, 1990; Schmidt & Hunter, 1983). However, it has been suggested that SDpcan alternatively be estimated by dividing supervisory estimates of SDy by supervisory estimates of the value of the performance of the 50th percentile worker, and that the mean estimate of SDP can then

Correspondence concerning this article should be sent to Michael K. Judiesch, Department of Management and Organizations, College of Business Administration, University of Iowa, Iowa City, Iowa 52242. 234

235

ESTIMATES OF DOLLAR VALUE

be multiplied by an objective estimate of average revenue value (?) to estimate SDy (Judiesch & Mount, 1988; Judiesch, Mount, & Schmidt, 1988). By examining, in this study, the relative accuracy of supervisory estimates of SDy and SDP, we sought to determine whether this modification of Schmidt et al.'s (1979) method yields more accurate estimates of SDy.

Evidence That Supervisors Underestimate Schmidt et al.'s (1979) Concept of the Dollar Value of the Output of the Average Employee Hunter and Schmidt (1982) noted that a profit-making firm must charge its customers an amount that exceeds the costs of labor, overhead, equipment, and materials. As this statement indicates, Schmidt et al.'s (1979) concept of the dollar value of the output of the average worker is the current yearly revenue obtained from the output that the average worker produces in combination with an average amount of the other factors of production. This concept implies that the current revenue value of the output of the average employee on a job (?) is equal to the current total yearly revenue from the output on that job divided by the number of employees on that job (Hunter, Schmidt, & Coggin, 1988). Because wages were approximately 57% of gross national product (GNP) in 1973, Hunter and Schmidt (1982) calculated the dollar value of output for an average employee as approximately 1.75 times mean wages (.57"'). However, Hunter and Schmidt (1982) overlooked the fact that GNP measures only final sales. Intermediate sales of materials are not included in GNP in order to avoid duplication. Only the value added to the final product by each firm contributes to GNP. As of 1985, wages were 49% of GNP, and compensation (wages plus benefits) was 59% of GNP. Hence, the value added by the average employee to the final sales price is approximately twice average wages (approximately 1.69 times average compensation). However, although variable material costs are excluded from a firm's contribution to GNP, variable material costs do affect the prices that a firm charges for its goods and services. Consequently, a firm's sales revenue is larger than its contribution to GNP. For 1985, corporate business receipts were $7,370 billion, and wages were $1,514 billion (U.S. Bureau of the Census, 1989). Thus, for an average firm, the sales revenue value of the output produced by the average employee using an average amount of the other factors of production is approximately five times the average wage. The ratio of sales revenue to wages varies widely across firms because of differences in capital intensity and variable material costs. In 1982, this ratio was 5.2 for manufacturing, 8.6 for retail trade, 20.8 for wholesale trade, and 2.7 for service industries (US. Bureau of the Census, 1982a, 1982b, 1982c, 1982d). The range of the revenue-to-wages ratio across individual jobs in individual firms is undoubtedly greater than the range of these industrywide ratios. In fact, it is theoretically possible that the average revenue value of employee output in one or more of the jobs in a firm in which overall revenue exceeds total wages could be less than the average wage on that job. Consequently, the extent to which these industrywide ratios reflect the ratios of the revenue value of output to wages for the specific jobs in previous SDy estimation studies is unknown. Nevertheless, these ratios strongly suggest that for the vast

majority of jobs an unbiased mean 50th percentile estimate of Schmidt et al.'s (1979) concept of dollar value should be considerably larger than average wages. However, the distributions of 50th percentile estimates obtained by Schmidt, Hunter, and coworkers have all been positively skewed, with the majority of supervisors providing a 50th percentile estimate that was close to average wages and a smaller group of supervisors providing much larger estimates. Other researchers have also reported positively skewed percentile distributions with either median or modal values for the 50th percentile that were within 15% of mean wages (cf. Bobko et al., 1983, Table 2; Burke & Frederick, 1984, Table 1; Eaton, Wing, & Mitchell, 1985; Greer & Cascio, 1987; Judiesch & Mount, 1988; Rich & Boudreau, 1987; Weekley et al., 1985). The large magnitudes of the ratios of revenue to wages shown in census data and the similarity between supervisors' 50th percentile estimates and mean wages appear to indicate that, although some supervisors interpret the estimation task in terms of revenue value, most supervisors make their percentile estimates on subjective scales that are based on the assumption that the value of an average employee is equal to wages, compensation (wages plus benefits), or possibly compensation plus a profit margin. The reason for the apparent downward bias in percentile estimates is unknown. However, the fact that many judges appear to equate average value with wages suggests these judges interpreted Schmidt et al.'s (1979) instructions as asking for a cost-based estimate of the value of the output uniquely attributable to workers rather than the value of the output produced in combination with the other factors of production. Schmidt et al.'s procedure asks judges to estimate the value of the products and services produced by employees at the three percentiles and suggests that as a guideline judges consider the cost of an outside firm providing these products and services. Apparently this cost-based guideline does not make it clear that judges are to estimate the value of output produced in combination with the other factors of production. For example, in applying their guideline to the job of computer programmer, Schmidt et al. assumed that the outside firm would use its data processing equipment and supplies in addition to its own programmer and that, consequently, the price charged by the outside firm would be sufficient to cover all of these costs plus a profit margin. The typical finding that some judges provide estimates considerably larger than wages suggests that these judges interpreted Schmidt et al.'s instructions in this manner. However, the close approximation of most judges' 50th percentile estimates to average wages indicates these judges took only labor costs into account in making their estimates. Consequently, the fact that many judges have provided 50th percentile estimates similar to average wages indicates that the use of alternative concepts of dollar value has resulted in a downward bias in the magnitude of estimates of the revenue value of the output of the 50th percentile employee. Implication of Different Concepts of Dollar Value

for Estimates of SDy Schmidt et al.'s (1979) method was designed to estimate utility in terms of the gain in revenue. However, as Boudreau (1991) and Hunter et al. (1988) noted, there is no single correct concept

236

M. JUDIESCH, F. SCHMIDT, AND M. MOUNT

of dollar value. Utility can be estimated in terms of revenue, pre- and posttax profit, or the savings in labor costs. Schmidt et al.'s (1979) method can be used to estimate SDy in terms of revenue, and other parameters can be added to the utility model to estimate utility in terms of profit (Boudreau, 1983, 1991; Hunter et al., 1988). Therefore, the fact that most supervisors apparently did not make their percentile estimates in terms of revenue has implications for the magnitude of SDy estimates and hence for the magnitude of the utility estimates obtained by all researchers who have incorporated Schmidt et al.'s method for estimating SDy into their utility estimation approaches. Therefore, the important issue of whether underestimation of percentiles due to the use of alternative interpretations of dollar value results in the underestimation of SDy is examined in detail in the following paragraphs. A downward bias in the mean estimate of SDy due to the use of alternative concepts of value does not necessarily imply that the actual SDy is underestimated. It is possible that other unknown judgmental errors exert an upward bias on judgments of SDy. Nevertheless, in the absence of evidence of an upward bias in judgments, a finding that mean estimates of SDy would have been larger had all of the judges made their percentile estimates in terms of revenue would support the hypothesis that mean estimates of SDy are lower than the actual revenue value of the standard deviation of job performance. Moreover, the hypothesis that supervisory estimates of SDy are biased downward applies only to Schmidt et al.'s (1979) method of estimating the standard deviation of the revenue value of performance. Our use of the term downward bias is not intended to imply that SDy should be estimated only in terms of revenue value, nor is it intended to imply that there may be a downward bias in alternative methods of estimating SDy (e.g., Cascio& Ramos, 1986; Raju, Burke, & Normand, 1990). Cascio and Ramos and Raju et al. have argued that the most appropriate measure of the average value of labor is the average cost of labor. This argument is based on the economic principle that a firm will hire labor up to the point at which the marker wage rate equals the value of the marginal product of labor. This position was also expressed by an anonymous reviewer, who argued that "to be consistent with accepted practice in the field of labor economics, the analysis should be in terms of marginal values." We disagree with this viewpoint for three main reasons. First, the purposes and conceptual bases of marginal productivity theory and utility analysis are very different. Marginal productivity theory is aimed at predicting and explaining how many workers a firm will hire. This theory assumes that (a) all factors of production except labor are held constant and (b) all workers hired are of equal value (Flanagan, Smith, & Ehrenberg, 1984). Utility analysis aims at the very different goal of predicting the benefit (e.g., increase in sales revenue, increase in profit, etc.) to be obtained by hiring the same number of workers with an improved selection method as would be hired with the current selection method. Thus, marginal productivity theory focuses on determining the organization's demand for labor, whereas utility analysis assumes a stable work force. The two models also differ in that the concept of individual differences in output is central to utility analysis but plays no role in marginal productivity theory. Second, the argument that SDy

must be estimated in terms of marginal value overlooks the fact that the utility estimate (At/) is already an estimate of the marginal value of hiring better workers. Depending on the metric in which Y is estimated (e.g., revenue, profit, output), At/ is an estimate of the total increase (i.e., marginal gain) in Y obtained by using an improved selection method (Schmidt et al., 1979). Consequently, it is not necessary to estimate Y in terms of its marginal value to make At/an estimate of the marginal value of improved selection. Third, equating the average value of labor with the average wage results in a utility estimate that is equal to the value of labor cost savings. As Hunter et al. (1988) demonstrated by way of example, this value understates both the gain in revenue and the gain in profit. The disagreement among researchers concerning the most useful concept of dollar value does not diminish the importance of the question of the judgment models used by supervisors. As noted earlier, the fact that most judges appear to use a wage-based model of supervisory value suggests that utility estimates that use Schmidt et al.'s (1979) SDy estimation method will underestimate the actual increase in sales revenue. Alternatively, the fact some judges make 50th percentile estimates that are much larger than average wages indicates that these utility estimates will be larger than the actual savings in labor costs. Thus, the use of multiple concepts of dollar value may result in utility estimates that do not accurately reflect any single concept of dollar value. However, an understanding of the judgment models used by supervisors may make it possible to design a more accurate method of estimating SDy and hence a more accurate method of estimating utility.

Alternative Models of Supervisory Scales of Estimates of Dollar Value Although many supervisors appear to use mean wages as a guideline for their 50th percentile estimate, it is not known what concept of differences in value they use to make their estimates of the other two percentiles. In many jobs, there is very little relationship between wages and output for individual employees (Bishop, 1987a, 1987b; Frank, 1984). Therefore, it cannot be assumed that these judges base all three of their percentile estimates on the distribution of wages. In the analysis that follows, we consider two alternative models of the decision rules that judges might use to make their three percentile estimates. As will be seen, a critical consideration is the fact that underestimation of the 50th percentile makes it impossible to accurately estimate simultaneously both SDy and SDP from supervisory estimates of the three percentile values. The first model (the additive constant theory) asserts that underestimation of the 50th percentile does not result in underestimation of SDy. However, dividing an accurate estimate of SDy by an underestimate of the 50th percentile causes the resulting estimate of SDP to be upwardly biased. According to the second model (the multiplicative constant theory), estimates of SDy are a constant percentage of the 50th percentile estimate. This implies that the use of concepts of dollar value other than sales revenue does not bias estimates of SDP. However, the theory that estimates of SDy are proportional to the 50th percentile estimates implies that underestimation of the 50th percentile causes estimates of SDy to be downwardly biased. Thus, resolution of the

237

ESTIMATES OF DOLLAR VALUE

question of how underestimation affects estimates of SDy and SDP is important to the accuracy of utility estimates in personnel selection and other personnel programs when Schmidt et al.'s (1979) procedure is used to estimate these parameters. The two theories also result in very different predictions about the psychometric characteristics of the estimates, and these conflicting predictions can be tested by using data from previous SDy estimation studies. To facilitate the comparison of the effect of underestimation of the 50th percentile on predictions made by the these two theories, we express individual estimates of the 50th percentile (Est. 50;) as

Est. 50; =

(2)

where a symbolizes the mean value of a.

Additive Constant Theory Any two scales that measure the same dimension can be described as transformations of each other. If the scales differ only by an additive constant, then the absolute difference between any two points on one scale is equal to the absolute difference between the same two points on the other scale. Consequently, if individual supervisors make percentile estimates on subjective scales that differ only by an additive constant from the sales revenue scale, estimates of SDy will be on the same scale because SDy is a difference score. This would be the case if different supervisors make their 50th percentile estimates on different scales but make their other two percentile estimates on the basis of the assumption that differences in value are approximately equal to differences in sales revenue. Thus, one hypothesis is that different supervisors first make their 50th percentile estimates on the basis of different concepts of dollar value but that all supervisors make their 85th percentile estimates on the basis of the assumption that the value of the output of the 85th percentile employee is equal to the estimated value of the output of the 50th percentile employee plus the increase in revenue obtained from the output of the 85th percentile employee over that obtained from the 50th percentile employee. The 15th percentile is then estimated by subtracting the estimate of the difference between the amount of revenue obtained from the output of employees at the 15th and the 50th percentiles from their 50th percentile estimate. Under these circumstances, given the usual assumption of normality, we can express individual estimates of the 85th percentile (Est. 8 5j) as

(3)

where a^Ty^ is as previously defined, SD^ is a constant equal to the true increase in revenue of the 85th percentile over the 50th percentile, and eu is the random error in estimating SD^. Similarly, the individual 15th percentile estimate (Est. 15;) is Est. 15; = aiTso - SD2 + e2i,

(4)

where SD^ is a constant equal to the true decrease in revenue of the 1 5th percentile in relation to the 50th percentile, and e2i is the random error in estimating SDy2. Hence, under the additive constant theory, the estimates of SDy for any single judge are

(1)

where Tio symbolizes the true dollar value of output as sold for the 50th percentile and a{ symbolizes the ratio of the individual estimate to the true value, that is, a{ = Est. 50-JTS0. The value of individual estimates of the 50th percentile is determined by both the amount of systematic downward bias due to estimating a concept other than sales revenue and the amount of random error. The effect of purely random error can be expected to average out over judges. Therefore, to the extent that downward bias exists, a< 1,

Est. 85, = a^ + SDy, + evo

Est. SDyli = (ajTVo + SD,i + «,i) - ajT-,0 = SDyt + elh

(5)

and

Est. SDy2, =fliTVo- (a,T50 - SDy2 + e2l) = SDy2 + e2i.

(6)

If this is the case, then the mean estimate ofSDy is an unbiased estimate of the true SDy.

Multiplicative Constant Theory When two scales differ by only a multiplicative constant, the absolute difference between any two points on one scale is equal to the constant multiplied by the difference between the two points on the other scale. In this case difference scores are not on the same scale and cannot be treated equivalently for purposes of data analysis. Therefore, if supervisors do not make percentile estimates in terms of sales revenue but instead provide estimates on a scale that differs from the revenue scale by a multiplicative constant that is less than one, the mean SDy estimate will be downwardly biased from the mean estimate that would have been obtained had supervisors estimated dollar value on the revenue scale. One possible method that supervisors could use to make dollar value estimates on scales that differ by only a multiplicative constant is that of magnitude estimation (Stevens, 1971). Thus, it can be hypothesized that supervisors first assign a number to the 50th percentile on the basis of their different concepts of the dollar value of output. They next estimate the 85th percentile so that the percentage or proportional difference between the 85th and the 50th percentile dollar value estimates is equal to their judgment of the percentage difference between the quantity of equivalent-quality output produced by employees at the 85th and the 50th percentiles. For example, if the 50th percentile is assigned an estimate of $20,000, an 85th percentile employee who is judged to produce 1.5 times as much as the 50th percentile employee is assigned an estimate of $30,000. The 15th percentile is then estimated by the same process, but with a ratio that is less than one. Under this theory, the 85th percentile estimate is equal to the 50th percentile estimate multiplied by the estimate of the ratio of the quantity of output produced by an employee at the 85th percentile to the quantity of output produced by an employee at the 50th percentile. This relationship may be expressed as

238

M. JUDIESCH, F. SCHMIDT, AND M. MOUNT

Est. 85; = (/c, +

Effect of Additive and Multiplicative Constants on Estimates of SDP

(7)

where A, is a multiplicative constant equal to the true ratio of the quantity of output at the 85th percentile to the quantity of output at the 50th percentile, ekn is the random error in estimating k,, and a{Tio is as previously defined. Because the true revenue value of output is proportional to the true quantity of output, the ratio of the true quantity of output produced at the two percentiles is equal to the ratio of the true revenue value of output at the 85th percentile (TK) to the revenue value of output at the 50th percentile. Substituting Tg5/Ti0 for /^, we see that

Est. 85, = (r85/rso + (8)

Similarly,

Est. 15; = (k2

ek2lT50),

(9)

where Tts is the true value of output at the 15th percentile, k2 = r15/r5o, ek2[ is the individual random error in estimating k2, and the other symbols are as previously defined. (Note that when the distribution of the value of employee output is normal or symmetrical, k2 = 2 — /q.) Consequently, the two individual estimates of SDy are Est. SZ>,H = a£TK + eknTio) - a{T SDy2) = r(50< 50 _15)). The interjudge correlation between estimates of the 50th percentile and esti-

240

M. JUDIESCH, F. SCHMIDT, AND M. MOUNT

mates of SDyl is psychometrically equivalent to the correlation between pretest scores (50) and gain scores (85 - 50), and the correlation between estimates of the 50th percentile and estimates of SDy2 is psychometrically equivalent to the correlation between posttest scores (50) and gain scores (50 - 15). Bartko and Pettigrew (1968) discussed these two forms of part-whole correlations (i.e., r(XY- x) and r(x x_ r>). Substituting 50 and 85 for A'and Y, their Equation 2 is '"(SO, SDyl)

Cov(50, 85) - Var(50) (20) 5Z)(50)[Var(50) + Var(85) - 2Cov(50, 85)]1/2 ' Following Bartko and Pettigrew's Equation 3, if we let v = Var(85)/Var(50), then Equation 20 becomes V" '"(SO, 85) ~ 1 '"(50,50,,) -

t>-2Vt>r ( 5 0 > 8 5 ) ] 1/2

(21)

Bartko and Pettigrew also demonstrated that r(x x-r> expressed in terms of their Equation 3 is equal to -r(jr> Y- x)- Thus, '"(50. 15) '(50, SDy2)

(22)

where v = Var(15)/Var(50). Bobko et al. (1983) and Burke and Frederick (1984) also obtained estimates of SDy for the interval between the 85th and 97th percentiles (SDy3). In the Appendix we show that a '"(SO, 97) ~ V^b r(50, 85)

1.00) and that the correlation of the 50th percentile with SDy2 is predicted to be positive (or zero if r(50 15) = 1.00). As Thorndike (1924) first demonstrated, random error exerts a negative bias on the correlation between pretest scores and difference scores and a positive bias on the correlation between posttest scores and difference scores. Because these are repeated measures made on multiple subjective scales, the correlation between percentiles can be predicted to be very high, resulting in a small negative value for /-(50i 5Dvi) and a small positive value for r(50 SDv2). These predictions are consistent with the prediction of the additive constant theory that the magnitude of SDy is not affected by the magnitude of the 50th percentile estimate (see Equations 5 and 6). Multiplicative constant theory. According to the multiplicative theory, the three percentile coefficients of variation are equal. When this is the case, the ratio of the standard deviations of adjacent percentile estimates is equal to the ratio of their means; that is, if SD^/MK = SDio/Mio, then A/85/M50 = SDK/ SDX, and if SD15/MIS = SDio/M50, then M^/M50 = SDi5/SDio. This implies that the ratio of the percentile variances, v, is equal to the square of the ratio of the percentile means, that is, Var85/ Var50 = (M85/M50)2 and Var,5/Var50 = (M15/M50)2. Thus, substituting the square of the ratio of percentile means for v in Equations 21 and 22 yields the following: '(50,

SDy,)

(M85/M50)r(50i 85) (M85/M50)2 - 2(M85/M50)r(50,85); 11/2

(27)

and

V '"(50, SDy,)

~

'

(23)

["a + Vb ~ ^ \ "a \ "b '(85, 97)J

(so, sDy

where ua = Var(97)/Var(50) and vb = Var(85)/Var(50). These equations indicate that the correlation between the 50th percentile and SDy is determined by the percentile intercorrelations and the ratio of the percentile variances. Therefore, because the two theories make different predictions about the ratio of the percentile variances, they make different predictions about the correlation between the 50th percentile and SDy. As we next demonstrate, for studies that have reported percentile means and percentile intercorrelations, it is possible to compare the predicted r(50i50 made by these theories with the actual r(50,2(15th-50th) Correlation between 50th percentile and SDP SD , (50th-85th) Relative variability o(SDyandSDp

Additive constant theory

Multiplicative constant theory

15th = 50th = 85th

15th < 50th < 85th

15th > 50th > 85th

15th = 50th = 85th

Small and negative Small and positive

Positive Large and positive

Large and negative Negative SDSDy SDSDa

Small and negative Small and positive SDWy SDSDa

242

M. JUDIESCH, F. SCHMIDT, AND M. MOUNT

for each of the predictions about the psychometric characteristics of estimates, sample-size weighted means resulted in identical conclusions.

Results Percentile standard deviations and coefficients of variation. The means, standard deviations, and coefficients of variation of the percentile estimates are presented in Table 2. The percentile standard deviations are given in column 3. For all of the studies, the standard deviation increased as the percentile magnitude increased. The FmM test for homogeneity of variance across the three percentiles was significant for each study (p < .01), indicating a significant difference between the variances of the three percentile estimates.1 Thus, the findings for Prediction 1 support the multiplicative constant theory. The percentile coefficients of variation are presented in the last column of Table 2. Within each study, the percentile coefficients of variation are very similar. The mean coefficients of variation for the 15th, 50th, and 85th percentiles are .76, .70, and .77, indicating that the percentile standard deviations are approximately a constant percentage of the mean of each percentile value. In the three studies in which the 97th percentile estimates were obtained, the coefficient of variation for this percentile is very similar to the other percentile coefficients of variation within each study. Hence, both the relationship between the percentile standard deviations and the relationship between the percentile coefficients of variation are consistent with the prediction made by the multiplicative constant theory that supervisors make their estimates of the 85th and the 15th percentiles by multiplying their 50th percentile estimates by their estimate of the relative difference between percentiles. Correlation between estimates of the 50th percentile and SDy. The predicted and actual correlations between estimates of the 50th percentile and estimates of SDy are presented in Table 3. The actual correlations are very similar to the correlations predicted by the multiplicative constant theory, but they differ greatly from the correlations predicted by the additive constant theory. The mean actual correlation between SDy and the 50th percentile is .71, which is almost identical to the mean correlation of .70 predicted by the multiplicative constant theory. The correlation between the 18 predicted correlations of the multiplicative constant theory and the actual correlations is .75, and the mean of the absolute differences between the predicted and actual correlations is .10. Only one correlation from one study —the correlation for the 15th-50th percentile interval in Rich and Boudreau (1987)—is significantly different from the correlation that was predicted by the multiplicative constant theory (p < .05). This indicates that the multiplicative constant model accurately predicted the actual correlations between SDy and the 50th percentile and provides additional support for the theory that supervisors make their 85th and 15th percentile estimates proportionate to their 50th percentile estimate. Correlation between estimates of the 50th percentile and SDP. The correlations between the 50th percentile and SDp are presented in Table 4. The mean correlation (.16) between SDP and the 50th percentile is not significantly different from zero, which is in agreement with the prediction of the multiplicative constant theory. Only 2 of the 13 correlations are statistically

different from zero, and these are significant in the opposite direction to that predicted by the additive constant theory. Relative variability ofSDy and SDP. Table 5 presents the comparison of the relative variability of estimates of SDy and SDp. The multiplicative constant theory receives additional support from the fact that all of the coefficients of variation of the SDp estimates are smaller than the coefficients of variation of the corresponding SDy estimates. The mean SDP coefficient of variation of 0.76 is 35% smaller than the mean SDy coefficient of variation of 1.17.

Discussion The results of Study 1 provide support for the hypothesis that supervisory estimates are based on processes similar to those predicted by the multiplicative constant theory. These findings indicate that a supervisor's estimate ofSDy is dependent upon the magnitude of his or her 50th percentile estimate. Most supervisors have provided 50th percentile estimates similar to mean wages. However, mean wages are much smaller than the sales value of the output of the average employee, indicating that the use of multiple interpretations of dollar value has resulted in a considerable amount of downward bias in estimates of SDy (and hence, other things equal, of estimates of utility). The equation recommended by Boudreau (1983) for estimating net utility is based on the assumption that SDy is estimated in terms of revenue value. Therefore, the results of this analysis also suggest that studies that have estimated utility in terms of profit would have produced larger estimates of profit if managers had used a revenue-based estimate of the 50th percentile rather than one based on average wages. The results of Study 1 also indicate that supervisory estimates of SDP are the same regardless of the concept of dollar value employed. Thus, despite the fact that the mean estimate of the revenue value of 50th percentile performance appears to be biased downward, the mean estimate ofSDy as a percentage of the 50th percentile appears to be unbiased. This suggests that the mean supervisory estimate of SDp obtained from the percentile estimates may be accurate enough to estimate utility in terms of the percentage increase in output, which in turn can be translated into an estimate of the reduction in payroll costs (Schmidt & Hunter, 1983; Schmidt, Hunter, Outerbridge, & Trattner, 1986). This finding also suggests that a reasonably accurate estimate of the revenue value of the output of the average employee (rso or Y) would make it possible to obtain a reasonably accurate revenue-based estimate of SDy. That is, the mathematical identity SDy = Y(SDy/Y) = Y(SDP) (30) suggests that it may be advantageous to first estimate Fand SDP separately and then multiply these estimates to estimate SDy

1 The Fmil test assumes that sample variances are independent. Consequently, the fact that the percentile variances are not independent indicates that the -Fmax value for the .01 percentage point is conservative.

243

ESTIMATES OF DOLLAR VALUE Table 2 Means, Standard Deviations, and Coefficients of Variation ofPercentile Estimates Study/percentile

N

Schmidt, Hunter, McKenzie, & Muldrow (1979) 15 50 85 McKenzie (1982) 15" 50 85 Bobko, Karren, & Parkington (1983) Yearly sales 15 50 85 97 Overall value 15 50 85 97 Burke & Frederick (1984) 15 50 85 97 Schmidt, Mack, & Hunter (1984) 15 50 85 Pearlman(1985) 15 50 85 Weekley, Frank, O'Connor, & Peters (1985) 15 50 85 Greer&Cascio(1987) 15 50 85 Rich & Boudreau (1987) 15 50 85 Judiesch & Mount (1988) 15 50 85

105

SD

V

$16,045 $25,896 $36,582

$14,487 $20,585 $33,840

0.90 0.79 0.93

— $23,378 $34,843

— $11,428 $17,605

— 0.49 0.51

$36,700 $96,000 $150,700 $180,600

$24,900 $83,200 $124,700 $139,700

0.68 0.87 0.83 0.77

$11,200 $16,000 $22,400 $26,200

$2,200 $7,800 $18,000 $22,000

0.20 0.49 0.80 0.86

$40,300 $75,200 $108,200 $134,700

$64,000 $101,900 $148,400 $179,600

1.50 1.36 1.37 1.33

$8,347 $13,531 $17,317

$2,869 $3,836 $5,437

0.34 0.28 0.31

$154,770 $381,190 $718,320

$139,530 $297,020 $573,010

0.90 0.78 0.80

$14,837 $27,537 $42,772

$13,148 $23,399 $43,799

0.89 0.85 1.02

$17,145 $31,979 $46,417

$14,633 $24,829 $38,443

0.85 0.78 0.83

$18,310 $33,924 $50,086

$16,797 $20,407 $34,469

0.92 0.60 0.69

$21,017 $26,870 $33,096

$8,805 $10,397 $13,336

0.42 0.39 0.40

$33,790 $67,934 $116,554 $113,833

$30,345 $53,335 $95,675 $113,767

0.76 0.70 0.77 0.99

60

13

13

26

114

134

110

29

29

83

Means 15 50 85 97 a

M

No estimate was obtained for the 15th percentile in this study.

(Judiesch & Mount, 1988; Judiesch et al, 1988). Later in this article, we present an objective method for estimating Y, As we will see, the accuracy of this estimate of Fdepends on the extent to which certain economic assumptions hold true. First, however, we address the question of whether estimates of SDP are accurate.

Study 2: An Empirical Test of the Accuracy of Estimates of SDP The results of Study 1 indicate that estimates of SDp are much less affected than SDy estimates by trie use of different concepts of dollar value. However, the presence of even a few estimates

244

M. JUDIESCH, F. SCHMIDT, AND M. MOUNT

Table 3 Correlations Between Estimates of the 50th Percentile and Estimates ofSDy Predicted r(50 s

Actual Study/percentile interval for SDy Additive Multiplicative r(50

both, may be smaller or larger than the coefficient of variation of the overall value of employee performance. However, Hunter et al. (1990) restricted their analysis of output-based estimates of SDp to those jobs in which they judged that the output measure represented the primary source of employee value.

SDy)

Schmidt, Hunter, McKenzie, &Muldrow(1979) a 50th-85thb 15th-50thc McKenzie (1982) 50th-85th Bobko, Karren, & Parkington(1983) Yearly sales 50th-85th 15th-50th 85th-97thd Overall value 50th-85th 15th-50th 85th-97th Burke & Frederick (1984) 50th-85th 15th-50th 85th-97th Schmidt, Mack, & Hunter (1984) 50th-85th 15th-50th Rich &Boudreau( 1987) 50th-85th 15th-50th Judiesch& Mount (1988) 50th-85th 15-50th Means 50th-85th 15th-50th 85th-97th Overall

Method -.20 .23

.48 .85

.61 .76

-.21

.51

.53

-.13 .23 -.17

.78 .97 .52

.75 .98 .35

-.10 .51 -.21

.81 .73 .63

.94 .96 .75

-.10 .24 -.18

.83 .90 .81

.84 .84 .78

-.21 .55

.30 .80

.43 .74

-.16 .37

.65 .85

.74 .58

-.14 .17

.50 .70

.58 .58

-.16 .33 -.19 .03

.61 .83 .65 .70

.68 .78 .63 .71

* The percentile and SDy intercorrelations for Schmidt et al. (1 979) are c reported in McKenzie (1982). SDy2. d SDyi

that differ from gross revenue by a negative additive constant will cause some upward bias in the mean estimate of SDp. In addition, it is possible that other unknown biases may cause the mean judgment-based estimate of SDP to be higher or lower than the actual coefficient of variation of the value of output. Thus, it is important to attempt to assess the accuracy of supervisory estimates ofSDp. Studies that have compared estimates of SDy for sales occupations with the actual standard deviation of sales output can be used to compare estimates of SDP with the actual coefficient of variation of sales. In addition, Hunter et al. (1990) recently presented SDP values based on counts of actual employee output on the job from many studies. This research found that SDP increases with increases in job complexity. Thus, a test of the accuracy of supervisory estimates of SDP can be obtained by comparing the mean supervisory estimates of SDP with the output-based mean SDP values for similar complexity jobs from Hunter et al. (1990). It is possible that the coefficient of variation of sales output, or physical counts of output, or

Supervisory estimates ofSDp. The overall mean supervisory estimate of SDP was computed for each of the 11 studies that were included in Study 1. For studies that provided tables with individual supervisory percentile estimates, it was possible to compute the actual mean supervisory estimate of SDP. However, for studies that reported only the mean estimate of SDy and the mean 50th percentile estimate, the ratio of these values is an approximation of the mean supervisory estimate of SDP. In general, the ratio of means is not necessarily equal to the mean of ratios. However, under the multiplicative constant theory, the expected value of the mean SDP estimate is equal to the expected value of the ratio of the mean estimate of SDy to the mean 50th percentile estimate. Thus, given an adequate sample size and errors of estimation that are random, this method will provide a value that is close to the correct value. This prediction was supported by results of a comparison of the actual mean supervisory estimate of SDP with the approximate mean supervisory estimate of SDP in studies that have provided the data needed to compute both values. The actual mean estimate of SDP and the approximate estimate of the mean SDP in these studies, respectively, are as follows: (a) for Bobko et al. (1983), yearly sales, 52.8% and 50.0%, and overall value, 25.0% and 31.0%; (b) for Burke and Frederick (1984), 42.0% and 43.0%; (c) for Rich and Boudreau (1987), 45.7% and 46.8%; and (d) for Judiesch and Mount (1988), 22.1% and 22.5%. With the exception of Bobko et al.'s (1983) overall value study, which had a sample size of 13, these values are very close. Because none Table 4 Correlations Between Estimates of the 50th Percentile and Estimates ofSDp Study/percentile interval for SDP Bobko, Karren, & Parkington (1983) Yearly sales 50th-85tha 15th-50thb 85th-97thc Overall value 50th-85th I5th-50th 85th-97th Burke & Frederick (1984) 50th-85th 15th-50th 85th-97th Rich & Boudreau (198 7) 50th-85th 15th-50th Judiesch & Mount (1988) 50th-85th 15th-50th Means 50th-85th 15th-50th 85th-97th Overall 'SO,,.

b

SDo2.

Actual r.(50,50,)

-.32 .39 -.31 .77 .88 .29 .04 .02 .00 .16 -.04 .06 .14 .14 .28 .01 .16

ESTIMATES OF DOLLAR VALUE

Table 5 Relative Variability of Estimates ofSDy and SDP Study/percentile interval for SDV and SD0 Bobko, Karren, & Parkington (1983) Yearly sales 50th-85th 15th-50th 85th-97th Overall value 50th-85th 15th-50th 85th-97th Burke & Frederick (1984) 50th-85th 15th-50th 85th-97th Rich &Boudreau( 1987) 50th-85th 15th-50th Judiesch& Mount (1988) 50th-85th 15th-50th Means 50th-85th 15th- 50th 85th-97th Overall

SDSDy/MSDy

SDSDp/MSDp

0.89 1.05 1.00

0.49 0.30 0.61

1.61

.49 .40

0.87 0.71 1.03

.58 .55 .33

1.00 1.22 1.17

1.04 0.92

0.72 0.75

0.69 0.63

0.55 0.45

1.16 1.05 1.24 1.17

0.73 0.68 0.94 0.76

of the studies for which an approximate estimate of SDP was computed used sample sizes smaller than 26 (Burke & Frederick, 1984) or 29 (Rich & Boudreau, 1987), the small difference between the actual and approximate estimates in these two studies suggests that the approximate estimates are close to the actual estimates. Output-based estimates of SDP. The output-based SDP values assigned to the jobs in the 11 studies were obtained from the reliabilitycorrected, output-based SDP values for job incumbents reported by Hunter et al. (1990). Unreliability due to intraindividual variability in day-to-day output inflates estimates of SDP computed from distributions of output (Hunter et al., 1990). However, the supervisory estimate of SDP is not computed with the usual formula for the coefficient of variation but is computed as the percentage difference between two percentile point estimates. The expected value of this difference is not biased upward by random measurement error (Judiesch, Schmidt, & Hunter, in press). Therefore, if supervisory estimates of SDP for job incumbents are accurate, they should be similar to the reliabilitycorrected, output-based SDP values for job incumbents. Five studies reported the mean and standard deviation of employee sales output, and the reliability-corrected output SDP for each study was found in Hunter et al.'s report. The jobs in the six remaining studies were assigned to low, medium, and high complexity levels in accordance with the system described by Hunter et al. This system assigned one job to the medium complexity level (skilled-crafts and technician jobs) and five jobs to the high complexity level (managerial and professional jobs). The output SDP value assigned to these jobs is the mean reliability-corrected SDP value reported by Hunter et al. for incumbents in jobs at the same level of complexity.

Results and Discussion An examination of Table 6 indicates that most of the mean supervisory estimates of SDP are quite similar to the corre-

245

sponding mean reliability-corrected estimate of SDp obtained from actual output data. Across the 11 studies, the mean of the supervisory estimates is 44.2%, which is very close to the actual output-based mean of 43.9%. The correlation between the two sets of values is .70. These findings indicate that, in general, supervisors have insight into the magnitude of relative (percentage) differences in employee performance. However, for two of the studies, the supervisory estimate of SDP is more than 10 percentage points larger than the actual output-based value. In Bobko et al.'s (1983) study, the mean estimate of SDp of 52.8% for yearly sales is somewhat higher than the actual yearly sales SDf of 37.5%. However, 60% of the work force had been at this job for less than one year, and employees with less than 3 months experience were excluded from the computation of the actual sales mean and standard deviation. Bobko et al. (1983) did not indicate whether supervisors were told to ignore the output of employees with less than 3 months experience. However, even if they were so instructed, the performance of these employees may still have influenced their judgments. The mean value obtained by Bobko et al. (1983) is also much smaller than that found in other studies of the sales output of insurance agents. For example, in S. H. Brown (1981), the mean reliability-corrected SDP values for first-year insurance agents in 12 companies with a total of 12,453 first-year employees ranged from 87.1% to 123.2%, with an overall mean of 108.5%. This indicates that there are tremendous differences in the performance of insurance agents during their first year on the job. Thus, the SDp value obtained from sales data in Bobko et al.'s (1983) study might have been much larger had the output of all of the employees been included in their analysis. The mean supervisory estimate for Greer and Cascio (1987) is also higher than the estimate of SDP obtained from sales data. The mean supervisory SDp estimates for the two intervals were within 2 percentage points of each other (SDpt = 45.2%; SDp2 = 46.4%). However, the actual sales-output distribution was positively skewed. The reported interquartile differences in actual sales output indicate that the ratio of the 85th-50th interval to the 50th percentile was approximately 45%, whereas the ratio of the 50th-15th interval to the 50th percentile was approximately 20%. Thus, the supervisory estimate for the 50th-85th percentile interval was quite accurate, but the estimate for the 15th50th percentile was much too large. This suggests that the large difference between the output of the top performers and that of the other employees may have caused supervisors to overestimate the amount of difference between the two lower performance levels. Although the mean estimates of SDP in these studies very closely match the values derived by Hunter et al. (1990), the distribution of SDP estimates is positively skewed in both Burke and Frederick's (1984) and Rich and Boudreau's (1987) studies. Most of the extreme values have a zero or negative 15th-percentile estimate, as well as an SDy estimate that is substantially larger than the mean SDy estimate. This suggests that a few supervisors may have provided percentile estimates that differed from sales revenue by an additive constant. Because such estimates cause estimates of SDP to be biased upward, the presence of these estimates suggests that the mean estimates of SDp in these studies would have been smaller if all of the judges had

246

M. JUDIESCH, E SCHMIDT, AND M. MOUNT

Table 6 Mean Estimates ofSDp Derived from Supervisory Judgments and Actual Output Data Study

Job

Schmidt, Hunter, McKenzie, & Muldrow (1979) McKenzie(1982) Bobko, Karren, & Parkington (1983) Yearly sales Overall value Burke & Frederick (1984) Schmidt, Mack, & Hunter (1984) Pearlman(1985) Weekley, Frank, O'Connor, & Peters (1985) Greer&Cascio(1987) Rich &Boudreau( 1987) Judiesch& Mount (1988)

Computer programmer Budget analyst Insurance counselor

Mean

Sales manager U.S. park ranger Sales executive Store manager Soft-drink salesman Computer programmer Nurse

Supervisory SD,

Output SDP

39.7a 49.0a

46.2" 46-.2b

52.8 25.0 42.0 33.2a 79.4a 51. l a 45.8" 45.7 22.1

37. 5C 37. 5C 41.3C 25.6" 76.3C 46.2" 33.3° 46.2" 46.2"

44.2

43.9

a

Approximate estimate computed by dividing the mean SDy estimate by the mean 50th percentile estimate. b Mean reliability-corrected output SDf for incumbents in jobs of similar complexity in Hunter, Schmidt, and Judiesch (1990). c Reliability-corrected, sales-output SDP for incumbents reported for that study by Hunter et al. (1990).

used the multiplicative constant model. This possible upward bias also would be expected to inflate the amount of variability in estimates of SDp. Therefore, we examined the effect of excluding these estimates on the mean and the standard deviation of SDP estimates. Dropping the two estimates of this type from Burke and Frederick's (1984) data reduced the mean supervisory estimate of SDP for that study from 42.0% to 31.6% and also reduced the standard deviation of these estimates from 44% to 17%. The deletion of the seven estimates of this type from Rich and Boudreau's (1987) data reduced the mean estimate of SDP for that study from 45.7% to 31.1% and reduced the standard deviation from 31% to 14%. (Recall that in Study 1 r(50 SDy2} in Rich and Boudreau's [1987] study was significantly different from the correlation that was predicted by the multiplicative constant theory. When the estimates with zero or negative values for the 15th percentile are excluded, the r(i0tSDy2) is .83 and the correlation predicted by the multiplicative constant theory is .79. This suggests that the remaining judges in Rich and Boudreau's [1987] study made estimates on scales that differed only by a multiplicative constant.) These reductions in the mean and the standard deviation of SDP estimates suggest that changing Schmidt et al.'s (1979) instructions to supervisors could perhaps result in estimates of SDP that are less variable and more accurate. Given that most judges seem inclined to use a multiplicative constant model (even when not told to do so), it might be a good idea to provide them with explicit instructions to do so. Instructions to supervisors could also be changed to take advantage of the fact that estimates of SDP do not have to be obtained from dollar value estimates. Because SDP is a scale-free measure, estimates of SDP can be computed from estimates of the quantity of equivalentquality output produced by employees at different levels of performance or, alternatively, from the number of employees at each level of performance required to produce a given level of output. Eaton et al. (1985) found that military officers placed greater credibility in the accuracy of their estimates when these

estimates were expressed in terms of the number of employees than when the estimates were expressed in terms of dollar value. Similarly, Judiesch and Mount (1988) found that nursing supervisors were significantly more confident in the accuracy of estimates expressed in terms of the number of patients that nurses at different levels of performance could care for at an equivalent level of quality than they were in the accuracy of their dollar-value percentile estimates. Thus, the ability to allow managers a choice of metrics may enhance the accuracy of supervisory estimates of SDP. Overall, the results of Study 2 indicate that the mean supervisory estimates of SDP tend to be similar to estimates of SDP computed from distributions of the quantity of actual employee output. This suggests that the mean supervisory estimate of SDP is accurate enough to estimate utility in terms of the percentage increase in output. The mean supervisory estimate of SDP can also be multiplied by the mean compensation of new employees to estimate the reduction in hiring costs (Judiesch et al., in press; Schmidt et al., 1986). These results also suggest that it may be possible to improve the accuracy of utility estimates by estimating SDy as the product of the mean supervisory estimate of SDP and a reasonably accurate objective estimate of mean revenue value (Y).

Estimating the Average Revenue Value of Output An objective estimate of mean value can be used to avoid the problem of judgmental error. In this section, we describe an objective method for estimating the mean revenue value of employee output. However, the fact that an estimate is objectively determined does not necessarily imply that it is accurate. Therefore, we delineate the economic assumptions that this estimate requires and examine research that addresses the robustness of these assumptions. As noted earlier, Schmidt et al.'s (1979) concept of dollar value implies that the mean revenue value of employee output (Y) is equal to total sales revenue divided by the total number of

ESTIMATES OF DOLLAR VALUE

employees. In general, the revenue value of output will include contributions from more than one job within an organization. In such cases, an approximate estimate of the average revenue value for a particular job can be calculated under the assumption that the contribution of each job to the total revenue of the firm is proportional to its share of the firm's total annual payroll. Thus, for a specific job (A): Job A value = total revenue • (Job A payroll/total payroll), (31) and

Y = Job A value/Job A number of employees.

(32)

Figures for total and job-specific payrolls can usually be easily obtained within an organization, making this method feasible from a practical point of view. The assumption that the contribution of the employees in a job to the total value of output is proportional to their share of total payroll derives from the principle of classical economic theory that in order to maximize profits a firm will employ labor to the point at which the marginal product of labor just equals the real wage (Flanagan et al., 1984). As noted by Alchian and Demsetz (1972), "The classic relationship in economics that runs from marginal productivity to the distribution of income implicitly assumes the existence of an organization, be it the market or the firm, that allocates rewards to resources in accord with their productivity" (p. 778). In replying to the objection that firms do not know what labor's marginal product is, Flanagan et al. (1984) stated that "whether employers can verbalize the profit-maximization conditions or not, they must instinctively know them in order to survive in a competitive environment" (p. 59). Frank (1984) and Bishop (1987a, 1987b) have presented evidence that wage differentials between employees in the same job are much smaller than productivity differentials. Their findings indicate that the assumption that individual employees are paid their marginal product is false. However, as Frank (1984) noted, the fact that individuals are not paid their marginal product does not contradict the principle that wages and marginal products have the same average value. The principle in economics that the average wage is equal to the average value of the marginal product of labor does not imply that the average value of what employees produce in combination with the other factors of production is equal to the average wage. In distinguishing between the value of the marginal product of labor and the value of the average product of labor, Miller and Meiners (1986) noted that "the wage rate is equivalent to the reduction in total revenues that the firm would suffer if one worker quit, other inputs remaining constant. However, the contribution of any worker (in conjunction with the nonlabor inputs) is equivalent to that worker's average physical product times the price of the output" (p. 515). Such factors as discrimination and firm-specific labor markets may limit the extent to which relative wages reflect the relative value of jobs (Becker, 1975; Rynes & Milkovich, 1986). If the relative wage of a job differs from its relative contribution to the value of the firm's output, the estimate of Y obtained from Equations 31 and 32 will deviate from the actual Y. However, utility estimation approaches that equate average value

247

with average wages or the market wage (e.g., Cascio & Ramos, 1986; Raju et al., 1990) are also subject to this same limitation. Furthermore, the fact that supervisory estimates of SDy appear to be influenced by wages suggests that this limitation also applies to utility estimation approaches that use Schmidt et al.'s (1979) method to estimate SDy. Although the presence of internal or firm-specific labor markets can be expected to limit the extent to which the average wage for a job equals the value of the average marginal product for that job, recent empirical studies indicate that firm-specific labor markets account for only a small component of life-cycle wage function. A study of the average wages in 359 jobs across 440 firms in the electronics industry (Leonard, 1989) found that, although average wages for the same job varied across firms, the standard deviation of firm effects was only 7%, indicating that wages were primarily determined by market forces common across firms. An analysis of the Michigan Panel Survey of Income Dynamics (J. N. Brown, 1989) found that firmspecific wage growth occurs almost exclusively during periods of on-the-job training and that there was no evidence that contractual considerations are an important source of firm-specific wage growth. J. N. Brown concluded that "wages increase with tenure primarily because productivity increases with tenure" (1989, p. 990). Several other recent studies have concluded that firm-specific tenure accounts for only a small component of life-cycle wage growth (Abraham & Farber, 1987; Altonji & Shakotko, 1987; Marshall & Zarkin, 1987; Ruhm, 1990; Topel, 1986). Thus, recent evidence supports the hypothesis that the relative wages of the jobs in a firm are approximately equal to their relative values. This suggests that for most jobs the method we propose will result in a reasonably accurate objective estimate of Y. In addition, the procedure could easily be modified by allowing senior executives to adjust the relative value of the job upward or downward if they judged that relative wages did not accurately reflect the relative values of jobs. Estimating Y, and hence SDy, from current financial data draws attention to the fact that linear regression-based estimates of future gains in revenue assume that the current values of the parameters in the utility equation will remain constant until all of the future gains are realized. (Contrary to the opinion of Steffy & Maurer [1988], Schmidt et al.'s [1979] instructions to estimate the yearly value to the company of the output produced refer to the current yearly value, not future yearly value. All regression-based predictions assume that current or past relationships, or both, can be used to predict future outcomes. This misinterpretation of Schmidt et al.'s instructions suggests that these instructions should be modified to specifically instruct supervisors to make their percentile estimates in terms of current annual value.) The important issue of the stability of test validities continues to be debated (Austin, Humphreys, & Hulin, 1989; Barrett & Alexander, 1989; Schmidt, Hunter, Outerbridge, & Goff, 1988). However, because Fis determined by the price and the quantity of average output, it is unlikely that SDy will remain precisely constant. Economic theory holds that the price of a product is determined by the price level at which the market supply and demand curves for that product intersect. The level of the supply curve is determined by the cost of the production input factors, whereas the level of the demand curve is determined by consumer income, consumer tastes, and the price of alternative

248

M. JUDIESCH, F. SCHMIDT, AND M. MOUNT

goods. Theoretically, any changes in these factors will cause a change in the equilibrium price for that product. In addition, economic theory suggests that the competitive environment in which a firm operates may affect the output price. In perfectly competitive markets, it is assumed that each producer's share of total product output is so small that changes in the amount of output produced by any one seller will have a negligible affect on the market price. Thus, all else equal, firms in perfectly competitive markets experience constant returns to scale (i.e., price per unit of output is unaffected by the quantity of output a firm produces), and a firm can produce and sell more output at the same price. However, for monopolies and oligopolies, economic theory holds that, all else equal, changes in the amount of output produced by each firm are large enough to change the equilibrium price, which implies that increases in output result in decreases in prices. This suggests that increases in output by firms in imperfectly competitive industries may result in decreases in SDy. However, although few industries are likely to be perfectly competitive, there is evidence that most industries are effectively competitive. Effectively competitive industries are characterized by fairly large numbers of firms, low entry barriers, and pricing close to marginal cost. Shepherd (1982) found that in 1980, 77% of U.S. national income was produced by industries that he classified as effectively competitive and that the degree of competition appeared to be increasing as a result of antitrust enforcement, increasing international trade, and government deregulation. Hence, for the majority of firms, economic theory indicates that increasing output will not significantly affect SDy. Nevertheless, for industries in which barriers to competition exist, utility estimates that fail to take into account the effects of increases in output on prices may overstate actual gains. In summary, there are numerous factors that may change SDy by altering the price or the quantity (or both) of worker output. Thus, the accurate estimation of the parameters in the utility equation does not guarantee that the utility estimate will be precisely accurate. However, this problem is inherent in the estimates of the future benefits of all of the non-risk-free investments that a firm makes. As Rao (1987) observed, if ex ante or estimated returns corresponded exactly to ex post or realized returns, then there would be no risk. Hence, the fact that the accuracy of utility estimates is limited by the extent to which SDy remains constant reflects the fact that there is a level of uncertainty associated with utility estimates (Rich & Boudreau, 1987). Thus, we conclude that under most conditions, a reasonable estimate of Yfor a given job can be computed with Equation 32. SDy can then be estimated as Y(SDP), which equals Y(SDy/Y), where SDP is computed from supervisory estimates of percentile values, as described earlier.

General Discussion We tested two alternative models of the decision-making process that supervisors could use to make their percentile estimates of the value of employee output to the organization. Several forms of evidence were found, all supporting the multiplicative constant theory, which posits that different supervisors

use different scales to make their estimate of the value of the performance of the average employee but that all supervisors use the same scale of percentage differences in value to produce their estimates of the other two percentiles. Supervisory estimates made in this manner result in estimates of SDy that, within the limits of random error, are a constant percentage of the estimate of the value of average performance. Most supervisors apparently have not interpreted the dollar value of output in terms of sales revenue but rather in terms of wages or salaries. This fact, combined with the finding that supervisory estimates of percentile values follow the multiplicative constant theory, has caused estimates of SDy to be downwardly biased. This suggests that, other things equal, the potential gains in sales revenue from improved personnel practices are larger than previous estimates of utility have indicated. The results of this study also indicate that supervisory estimates of the coefficient of variation of job performance, SDP, are more accurate than supervisory estimates of SDy. This conclusion is supported by (a) the fact that judges appear to use a ratio model in making their estimates and (b) the fact that supervisory estimates of SDP are similar to empirical outputbased SDp values. The accuracy of supervisory estimates of SDp is also supported by the fact that they are less variable than estimates of SDy. Although lower variability does not prove that estimates of SDp are more accurate than estimates of SDy, it is consistent with this conclusion. It also indicates that fewer judges are needed to obtain reliable estimates. In this article, we described a method for obtaining an objective estimate of the mean revenue value of employee output (Y) from company payroll and revenue data. Considerable attention was given to the many economic assumptions that underlie both this method of estimating average value and the general linear regression-based utility equation. This estimate of F can be multiplied by the mean supervisory estimate of SDP to obtain an estimate of SDy. The fact that estimates of SDP are similar to output-based SDP values suggests that this method will result in a reasonably accurate estimate of SDy. An additional advantage of estimating SDy from estimates of SDP is that it is not necessary that estimates of SDp be obtained from dollar value estimates. Finally, we note that the results of this study may have more general implications for research in the area of human judgment and decision making. Roth (in press) noted that Schmidt et al.'s (1979) method is similar to fractile estimation in that both involve percentile point estimates and appear to be subject to the anchoring and adjustment heuristic. Block and Harper (1991) concluded that judges used a ratio model in making their fractile estimates, suggesting that the anchoring and adjustment heuristic does not simply involve adding and subtracting a constant number from the anchoring point estimate. Our conclusion that judges use a ratio model in making their percentile estimates of dollar value is consistent with the findings of Block and Harper (1991) and suggests that ratio models play an important role in judgments of uncertain quantities.

References Abraham, K. G., & Farber, H. S. (1987). Job duration, seniority, and earnings. American Economic Review, 77, 278-297.

ESTIMATES OF DOLLAR VALUE Alchian, A. A., & Demsetz, H. (1972). Production, information costs, and economic organization. American Economic Review, 62, 777795. Altonji, J. G., & Shakotko, R. A. (1987). Do wages rise with job seniority? Review of Economic Studies, 54, 437-459. Austin, J. T, Humphreys, L. G, & Hulin, C. L. (1989). Another view of dynamic criteria: A critical reanalysis of Barrett, Caldwell, and Alexander. Personnel Psychology, 42, 583-596. Barrett, G. V, & Alexander, R. A. (1989). Rejoinder to Austin, Humphreys, and Hulin: Critical reanalysis of Barrett, Caldwell, and Alexander. Personnel Psychology, 42, 583-596. Bartko, J. J., & Pettigrew, K. D. (1968). A note on the correlation of parts with wholes. American Statistician, 22(October), 41. (See also Erratum, 22 [December], 21.) Becker, G. (1975). Human capital (2nd ed.). New York: National Bureau of Economic Research. Bishop, J. (1987a). Information externalities and the social payoff to academic achievement (Paper 87-06). Ithaca, NY: Cornell University, Center for Advanced Human Resource Studies. Bishop, J. (1987b). The recognition and reward of employee performance. Journal of Labor Economics, 5(No. 4, Part 2), S36-S56. Block, R. A., & Harper, D. R. (1991). Overconfidence in estimation: Testing the anchoring-and-adjustment hypothesis. Organizational Behavior and Human Decision Processes, 49,188-207. Bobko, P., Karren, R., & Kerkar, S. P. (1987). Systematic research needs for understanding supervisory estimates of SDy in utility analysis. Organizational Behavior and Human Decision Processes, 40, 69-95. Bobko, P., Karren, R., & Parkington, J. J. (1983). Estimation of standard deviations in utility analyses: An empirical test. Journal of Applied Psychology, 68, 170-176. Boudreau, J. W (1983). Economic considerations in estimating the utility of human resource productivity improvement programs. Personnel Psychology, 36, 551-576. Boudreau, J. W (1991). Utility analysis for human resource management decisions. In M. D. Dunnette (Ed.), Handbook of industrialorganizational psychology (Vol. 2, 2nded., pp. 621-745). Palo Alto, CA: Consulting Psychologists Press. Brown, J. N. (1989). Why do wages increase with tenure? On-the-job training and life-cycle wage growth observed within firms. American Economic Review, 79,911 -991. Brown, S. H. (1981). Validity generalization and situational moderation in the life insurance industry. Journal of Applied Psychology, 66, 664-670. Burke, M. J., & Frederick, J. T. (1984). Two modified procedures for estimating standard deviations in utility analysis. Journal of Applied Psychology, 69, 482-489. Campbell, J. P. (1990). Modeling the performance prediction problem. In M. D. Dunnette (Ed.), Handbook of industrial-organizational psychology (Vol. 1, 2nd ed., pp. 687-732). Palo Alto, CA: Consulting Psychologists Press. Cascio, W F, & Ramos, R. A. (1986). Development and application of a new method for assessing job performance in behavioral/economic terms. Journal of Applied Psychology, 71, 20-28. Eaton, N. K., Wing, H., & Mitchell, K. (1985). Alternative methods of estimating the dollar value of performance. Personnel Psychology, 38, 27-40. Firebaugh, G, & Gibbs, J. P. (1985). User's guide to ratio variables. American Sociological Review, 50, 713-722. Flanagan, R. J., Smith, R. S., & Ehrenberg, R. G. (1984). Labor economics and labor relations. Glenview, IL: Scott, Foresman. Frank, R. H. (1984). Are workers paid their marginal products? American Economic Review, 74, 549-571. Greer, O. L., & Cascio, W F. (1987). Is cost accounting the answer? Comparison of two behaviorally based methods for estimating the

249

standard deviation of job performance in dollars with a cost-accounting approach. Journal of Applied Psychology, 72, 588-595. Hunter, J. E., & Schmidt, F. L. (1982). Fitting people to jobs: The impact of personnel selection on national productivity. In M. D. Dunnette & E. A. Fleishman (Eds.), Human performance and productivity: Human capability assessment (Vol. 1, pp. 233-284). Hillsdale, NJ: Erlbaum. Hunter, J. E., Schmidt, F. L., & Coggin, D. (1988). Problems and pitfalls in using capital budgeting and financial accounting techniques in assessing the utility of personnel programs. Journal of Applied Psychology, 73, 522-528. Hunter, J. E., Schmidt, F. L., & Judiesch, M. K. (1990). Individual differences in output variability as a function of job complexity. Journal of Applied Psychology, 75, 28-42. Judiesch, M. K., & Mount, M. K. (1988, April). A comparison of supervisory estimates ofSDr andSDP. Paper presented at the Third Annual Conference of the Society for Industrial/Organizational Psychology, Dallas, TX. Judiesch, M. K., Mount, M. K., & Schmidt, F. L. (1988). A comparison of supervisory estimates ofSDy and SDP. Unpublished manuscript. Judiesch, M. K., Schmidt, F. L., & Hunter, J. E. (in press). Has the problem of judgment in utility analysis been solved? Journal of Applied Psychology. Kasarda, J. D., & Nolan, P. D. (1979). Ratio measurement and theoretical inference in social research. Social Forces, 58, 212-227. Leonard, J. S. (1989). Wage structure and dynamics in the electronics industry. Industrial Relations, 28, 251-275. MacMillan, A., & Daft, R. L. (1979). Administrative intensity and ratio variable: The case against definitional dependency. Social Forces, 58, 228-248. Marshall, R. C., & Zarkin, G. A. (1987). The effect of tenure on wage offers. Journal of Labor Economics, 5, 308-330. McKenzie, R. C. (1982). A new procedure for estimating the standard deviation of job performance (OPRD Report 82-4). Washington, DC: U.S. Office of Personnel Management. Miller, R. L., & Meiners, R. E. (1986). Intermediate economics (3rd ed). New York: McGraw-Hill. Pearlman, K. (1985, August). Development of a dollar criterion for highlevel sales jobs. Paper presented at the 93rd Annual Convention of the American Psychological Association, Los Angeles. Raju, N. S., Burke, M. J., & Normand, J. (1990). A new approach for utility analysis. Journal of Applied Psychology, 75, 3-12. Rao, R. K. S. (1987). Financial management: Concepts and applications. New York: Macmillan. Reilly, R. R., & Smither, J. W (1985). An examination of two alternative techniques to estimate the standard deviation of job performance in dollars. Journal of Applied Psychology, 70,651 -661. Rich, J. R., & Boudreau, J. W (1987). The effects of variability and risk in selection utility analysis: An empirical comparison. Personnel Psychology, 40. 55-84. Roth, P. L. (in press). Research trends in judgment and their implications for the Schmidt-Hunter global estimation procedure. Organizational Behavior and Human Decision Processes. Ruhm, C. J. (1990). Do earnings increase with job seniority? Review of Economics and Statistics, 72, 143-147. Rynes, S. L., & Milkovich, G. T. (1986). Wage surveys: Dispelling some myths about the market wage. Personnel Psychology, 39,71 -90. Schmidt, F. L., & Hunter, J. E. (1983). Individual differences in productivity: An empirical test of estimates derived from studies of selection procedure utility. Journal of Applied Psychology, 68, 407-414. Schmidt, F. L., Hunter, J. E., McKenzie, R., & Muldrow, T. (1979). The impact of valid selection procedures on workforce productivity. Journal of Applied Psychology, 64, 609-626. Schmidt, F. L., Hunter, J. E., Outerbridge, A. N., & Goff, S. (1988). Joint relation of experience and ability with job performance: Test of three hypotheses. Journal of Applied Psychology, 73, 46-57.

250

M. JUDIESCH, F. SCHMIDT, AND M. MOUNT

Schmidt, F. L., Hunter, J. E., Outerbridge, A. N, & Trattner, M. H. (1986). The economic impact of job selection methods on size, productivity, and payroll costs of the federal work force: An empirically based demonstration. Personnel Psychology, 39, 1-29. Schmidt, F. L., Mack, M. J., & Hunter, J. E. (1984). Selection utility in the occupation of U.S. Park Ranger for three modes of test use. Journal of Applied Psychology, 69, 490-497. Shepherd, W G. (1982). Causes of increased competition in the US. economy, 1939-1980. Review of Economics and Statistics, 64, 613626. Steffy, B. D., & Maurer, S. D. (1988). The dollar-productivity impact of the human resource function: Conceptualization and measurement. Academy of Management Review, 13, 271-286. Stevens, S. S. (1971). Issues in psychophysical measurement. Psychological Review, 78, 426-450. Thorndike, E. L. (1924). The influence of the chance imperfections of measures upon the relation of initial score to gain or loss. Journal of Experimental Psychology, 7, 225-232.

Topel, R. (1986). Job mobility, search, and earnings growth: A reinterpretation of human capital earnings functions. In R. G. Ehrenberg (Ed.), Research in labor economics (Vol. 8, pp. 199-223). Greenwich, CT: JAI Press. US. Bureau of the Census. (1982a). Census of manufacturing industries: 1982. Washington, DC: US. Department of Commerce. U.S. Bureau of the Census. (1982b). Census of retail trade: 1982. Washington, DC: U.S. Department of Commerce. U.S. Bureau of the Census. (1982c). Census of service industries: 1982. Washington, DC: U.S. Department of Commerce. U.S. Bureau of the Census. (1982d). Census of wholesale trade: 1982. Washington, DC: U.S. Department of Commerce. U.S. Bureau of the Census. (1989). Statistical abstract of the United Sto«(109thed.). Washington, DC: U.S. Department of Commerce. Weekley, J. A., Frank, B., O'Connor, E. J., & Peters, L. H. (1985). A comparison of three methods of estimating the standard deviation of performance in dollars. Journal of Applied Psychology, 70, 122126.

Appendix Derivation of r(SO Cov(50, 97) - Cov(50, 85) Var(85) - 2Cov(97, 85)]'/2

(Al)

[Var(97) + Var(85) - 2SZ>97S£>85r(97,85)]"2 [Var(97)/Var(50) + Var(85)/Var(50) ,7,85))/Var(50)]"2

(A-^ (A4)

Let v, = Var(97)/Var(50) and vb = Var(85)/Var(50). Then Equation A4 becomes '(50, SO,,)

„ 1/2, _ „ 1/2.. r a (50, 97) ^b '(50,85) , v _ -, v 1/2 1/2 U/2

V

/

(V* + \>

*- a

_

(A2)

5£»50fVar(97) + Var(85) -

_

Under the additive constant theory, Var(50) = Var(85) = Var(97). Therefore, u, = t)b = 1. Thus, Equation A5 becomes

vb

r(97 85))

(A5)

'(50. 50,,,) •

r

(50, 97) ~ r(iO, 85)

(2-2r (97 , !5) )''

(A6)

Under the multiplicative constant theory, the ratio of percentile variances is equal to the square of the ratio of percentile means. Therefore, u, = (M91/Mio)2 and «b = (A/85/M50)2. Hence, Equation A5 becomes '(50, 50,,)

(M,7/Mso)r(>0i 97) A7 5o)2 - 2(A/97/M50)(A/85/M50)/-(85,97)] i/S • ( >

Received September 7,1989 Revision received December 2,1991 Accepted December 6,1991