Two New Procedures for Studying Validity Generalization

Copyright 1983 by the American Psychological Association Inc

Journal of Applied Psychology 1983, Vol 68, No 3, 382-395

Two New Procedures for Studying Validity Generalization Michael J. Burke

Nambury S. Raju

Psych Systems, Inc.. Baltimore, Maryland

Illinois Institute of Technology

The recent significant work of Schmidt and Hunter and their colleagues, and of Callender and Osbura, has resulted in several procedures for testing whether validity is generahzable Monte Carlo studies based on these procedures have thus far yielded good support, however, the degree of accuracy of these procedures in estimating the appropriate population parameters still has room for improvement In this article, two new procedures, which are based on less restrictive assumptions than those used by the previous investigators, are proposed for studying validity generalization Several Monte Carlo studies were earned out to test the accuracy of the new procedures in estimating population true validity mean and vanance and to compare these estimates with those of the other currently available procedures In general, the results indicated that one of the new procedures provided slightly more accurate estimates of the mean and vanance of population true validity than the procedures of Schmidt, Hunter, and their colleagues or of Callender and Osburn From a practical point of view, however, the estimates from the vanous procedures are quite comparable

Thousands of validity studies are conducted annually to determine the predictive effectiveness of numerous tests for selecting, classifying, and placing individuals in organizations. Usually the results for a single test m an organization are viewed in isolation with little, if any, regard for results obtained at other times or in other organizations (Linn, Harnisch, & Dunbar, 1981) A primary reason is that most personnel psychologists have traditionally believed that test validities are situationally specific. Contributing to this belief is the fact that for over 40 years validity coefficients have averaged below .30. More importantly, the observed validity coefficients have varied considerably from study to study, even when the jobs and tests studied appeared to be similar or the same (Ghiselli, 1966). According to Guion (1976), this inability to generalize validities has made it impossible for personnel psychologists to develop the general principles and theories that

The authors would like to express their appreciation to John C Callender and Frank L Schmidt for the many constructive and detailed comments on an earlier version of this article We also thank Jack E Edwards, Robert L. Linn, and Jeffrey A Slinde for their comments Request for reprints should be sent to Nambury S Raju, Department of Psychology, Illinois Institute of Technology, Chicago, Illinois 60616

are necessary to take the field of industrial psychology from a mere technology to a science. Recently, Schmidt, Hunter, and their colleagues (Pearlman, Schmidt, & Hunter, 1980; Schmidt, Gast-Rosenberg, & Hunter, 1980; Schmidt & Hunter, 1977; Schmidt, Hunter, Pearlman, & Shane, 1979) have offered an alternative explanation for the considerable between-study variation m observed validity coefficients. They suggested that much of the variance in validity outcomes for similar test-job combinations is due to eight sources or artifacts- sample size, criterion reliability, test reliability, range restriction, typographical and computational errors, criterion contamination, factor structure of tests, and factor structure of criterion measures. According to them, the factor structure of criterion measures should be the sole basis for concluding situational specificity. They also contend that much of the variance in the validity coefficients is due to the first seven artifacts and, therefore, validity is often generalizable. Using the four artifacts that are presently quantifiable (sample size, criterion reliability, test reliability, and restriction of range), Schmidt and Hunter (1977) proposed a procedure for determining how much of the variation in observed validity coefficients is due to these artifacts. The latest versions of

382

383

VALIDITY GENERALIZATION

Schmidt and Hunter's original procedure are referred to as the "noninteractive" (or "additive") procedure (Pearlman et al., 1980) and the "interactive" procedure (Schmidt et al., 1980). Recently Callender and Osburn (1980) proposed a different procedure, which is referred to as the "independent multiplicative" procedure. These three procedures correct for the four quantifiable artifacts mentioned above Recently, several Monte Carlo studies have been conducted to determine the accuracy as well as the similarity of the three procedures (Callender & Osburn, 1980, 1982, Callender, Osburn, Greener, & Ashworth, 1982) Although the results from these studies have shown that the three procedures yield estimates that are quite similar, their estimates are slightly inaccurate. Hence, there is a need to improve the existing procedures or to develop new ones for studying validity generalization Recently, Raju (Note 1) proposed two new procedures for assessing validity generalization under assumptions that are less restrictive than those underlying the noninteractive, interactive, and independent multiplicative procedures. Following Linn and Dunbar (Note 2), the two new procedures will hereafter be referred to as Taylor Series Approximation 1 (TSA 1) and TSA 2 The major purpose of this article is to empirically compare TSA 1 and TSA 2 with the noninteractive, interactive, and independent multiplicative procedures using the Monte Carlo methods. To gam a proper understanding as to why the assumptions of TSA 1 and TSA 2 are less restrictive, it would be advantageous to briefly review the correlation model upon which the noninteractive, interactive, and independent multiplicative procedures and TSA 1 and TSA 2 are based New Procedures Using the standard psychometric formulas (Lord & Novick, 1968), the observed correlation (r) between x and y in a sample can be written as \/2r 1/2

(1) Here, r w and r^ are the reliabilities of criterion and predictor, respectively; u is the

ratio of restricted standard deviation of x to unrestricted standard deviation of x; and e is the sampling error. The correlation model given in this equation incorporates the effect of criterion unreliability, test unreliability, range restriction, and sample size on the observed correlation. As shown by Raju (Note 3) the estimates of residual variance in the noninteractive and independent multiplicative procedures can be derived from the above equation under appropriately stated assumptions Following Callender and Osburn (1980), Equation 1 can be rewritten as r = pabc + e, [/2

(2)

12

where a = ryv , b = r^ , and c = u/(l + (u2 — i)p2rnrxx)i/2. The variance of r(VT) is given by Vr = V(pabc + e) = V{pabc) + Ve (3)

assuming that the errors (e) are uncorrelated with the product pabc Strictly speaking, e is correlated with pabc (Hedges, Note 4), but according to Linn and Dunbar (Note 2) its effect is likely to be very minimal. The above equation is not very useful in practice because the information concerning a, b, and c is generally not available Therefore, m further simplifying Equation 3, Callender and Osburn assumed that p, a, b, and c were independent Although a good case can be made for the independence of p, a, and b, it is difficult to accept that c is independent of p, a, and b when they are functionally related. A less restrictive assumption would have been that u and p, u and a, and u and b were independent of each other. TSA 1 and TSA 2 were derived to take into account the functional dependence of c on p, a, and b. The assumptions associated with the two procedures are for TSA 1, p, a2{ryv), b^r^), and u are pairwise independent, for TSA 2, p, a, b, and u are pairwise independent. Obviously, the two sets of assumptions are different with respect to a and b TSA 1 involves assumptions concerning the reliabilities of the criterion and predictor, whereas assumptions in TSA 2 involve the square roots of the same quantities We feel that the use of a2(r>v) and 62(rxx) in the simplification of Equation 3 is much more natural than the use of a and b. We do, however, offer solutions based on both

384

NAMBURY S RAJU AND MICHAEL J BURKE

sets of assumptions. Because the research of Schmidt, Hunter, and their colleagues and of Callender and Osburn involved assumptions concerning a and b rather than a2 and b2, TSA 2 will offer a more direct comparison with their procedures The true mean validity and variance estimation procedures for TSA 1 and TSA 2 are presented in Table 1. In addition, the standard statistical formulas (Johnson & Kotz, 1970; Pearlman et al., 1980) employed by each procedure for estimating mean observed validity, observed validity variance, and variance of sampling error are shown in Table 1 The derivations of the two new procedures are shown in Appendix A. The true mean and variance estimation equations for the noninteractive, interactive, and independent multiplicative procedures are given m Table 2 The reader is referred to Schmidt et al. (1980) for a description of the noninteractive and interactive procedures and to Callender and Osburn (1980) for a description of the independent multiplicative procedure. Estimation of Vp and Mp Either TSA 1 or TSA 2 may be used to estimate the mean and variance of p To use these procedures, however, one must have information concerning the means and variances of r, e, a, a2, b, b2, and u. Although the means and variances of r and e are readily computable in practice, such is not the case with respect to a, a2, b, b2, and u Because criterion reliability (a2), test reliability (b2), and restriction of range (u) are rarely reported for individual validity studies in practice (Schmidt & Hunter, 1977), the means and variances of these quantities must be estimated in order to determine whether validity generalization is possible Recently, Schmidt and Hunter (1977) and Pearlman et al (1980) proposed the use of certain hypothetical distributions for a2, b1, and u when these values are not available from individual validity studies; they also offered a rationale for recommending these distributions The hypothetical distributions are given in Table 3. The means and variances of these distributions are as follows Ma = .7686, Va = .0093, Mai = 6000, Vai = .0214, Mb = .8943, Vb = .0022, Mb2 = .8020, Vh2 = .0066, Mu =

.5945, and Vu= 0146 With the help of these means and variances, Mp and Vp can be estimated in TSA 1 and TSA 2. It should be noted that the same hypothetical distributions are also used m the implementation of the noninteractive, interactive, and multiplicative procedures Method A computer simulation study was conducted to determine the accuracy of the previously described validity generalization procedures for estimating the mean and variance of true validities in three special cases These cases were suggested by Callender and Osburn (1980) for assessing any newly proposed validity generalization procedures For each of the three cases, the computer simulation technique generated restricted, attenuated bivanate validity coefficients from known distributions of true validities The specific procedure for generating observed validity coefficients in each case is described below

Case 1 For the first case, true validity was held constant and criterion reliability, predictor reliability, and range restriction effects were varied Values of criterion and predictor reliability and range restriction effects were sampled from the distributions in Table 3 The procedure for producing observed validity coefficients initially involved crossing the 100 values of criterion reliability, 100 values of predictor reliability, and 100 values of range restriction effects This resulted in 1,000,000 combinations of predictor, criterion reliability, and range restriction effects Each combination was thereafter applied to the constant true validity of 1, which resulted in 1.000,000 observed, attenuated, restricted validity coefficients Then thefivevalidity generalization procedures were individually applied to this distribution of observed validity coefficients to assess their comparative accuracy in estimating the true validity of 1 and true validity variance of 0 This procedure was repeated for each true validity. 2 to 9

Case 2 For the second case, criterion reliability, predictor reliability, and range restriction effects were held constant while true validities were allowed to vary A distribution consisting of 100 true validities was constructed by taking each of the values from 00 to 99 in intervals of 01 The values for the artifactual effects were chosen by taking the upper limit (that is, the value of 1 0 that indicated lack of the artifactual effect), average, and lowest value of the respective distributions from Table 3 These constant values were 1 0, 60, and 30 for criterion reliability, 1 0, 80, and 50 for predictor reliability, and 1 0, 603, and 411 for range restriction effects The rationale for choosing these values was to obtain a broad range of combinations of artifactual effects as well as provide a replication of Callender and Osburn's (1980) Case 2 in a subset of the trials The values for the three statistical

385

VALIDITY GENERALIZATION Table 1 True Mean and Variance Estimation Procedures for Taylor Series Approximations TSA 1

(TSAs) 1 and 2

TSA2

1 Collect data on validity coefficient (r) and sample size for a set of studies

1 Collect data on the validity coefficient (r) and sample size for a set of studies

2 Calculate the mean criterion reliability (Mai), mean predictor reliability (Mb*), and the mean range restriction effect (Mu, see Table 3)

2 Calculate the mean criterion reliability (Ma), predictor reliability (Mb) and range restriction (Mu, see Table 3) '

3 Calculate the mean observed validity coefficient (Mr) weighted for sample size

3 Calculate the mean observed validity coefficient (Mr) following Step 3 of TSA 1

J.N,r, N

'

where A', is the number of subjects in study i, r, is t the validity coefficient for study / and A' = 2 N, 4 Calculate the estimated mean true validity coefficient (Mf) M =•

4 Calculate the estimated mean true validity coefficient (Mf)

M)(\-

'

MaMb(Mu +

M2(\-Ml))112

5 Calculate the variance of criterion reliability (Vai), predictor reliability (Vbi), and the effect of range restriction (\ „, see Table 3)

5 Calculate the variance of the effect of criterion reliability (I' a ), predictor reliability (Vb), and range restriction (Vu, see Table 3)

6 Calculate the variance of sampling error (Vc)

6 Calculate the variance of sampling error (Ve) by following Step 6 of TSA 1

y N,(\^ 'I! N,~ 1 v,=Ar

7 Compute the variance of observed validity coefficients (Vr)

7 Calculate the variance of observed validity coefficients (Vr) by following Step 7 of TSA 1

M2r

Vr = N 8 Calculate A, B, C, and D

8 Compute E, F, G, and H

r M,(\_ Mr_ ~M +

Ml)

M, M}(\ - Ml) ~ Mo+ M,Ml '

Ml)\

1 I Mr 2 \Ma2

MalM •2 J>

2 \Ma

1 / Mr 2W^

- •

°~

-

V

'

~

V

'

~

}'

I (ML ~ 2\M .Mbb ' H=

9 Compute the estimated variance of true validity coefficients (fj V

MaMl

=

Mr- M Mu

D

+

•g2K»2

~

A2

C

l

V

^

~

D

2

V

Mr- Mr Mu

9 Compute the estimated variance of true validity coefficients (Vp)

"

y =

Vr-\\-F2Va-C2Vb-H2Vu _

386

NAMBURY S RAJU AND MICHAEL J. BURKE

Table 2 Estimation Equations for True Validity Mean and Variance Procedure

Equation True mean validity estimation

Noninteractive, interactive, and independent multiplicative

Mr

True validity variance estimation -

Noninteractive (additive)

Vr-Vt-M2p(Va K-

Interactive y

Independent multiplicative

_ Vr-Ve-

artifacts were completely crossed, producing 27 combinations of criterion and predictor reliability and range restriction effects Each combination was separately applied to the hypothetical true validity distribution This resulted in 100 attenuated, restricted validity coefficients for each analysis For each of the 27 distributions of 100 observed validity coefficients, the five validity generalization procedures were applied to estimate the mean and variance of true validities

Case 3 The third case that was examined varied all four factors true validity, criterion reliability, predictor reliabil-

MlVc))

Ve-M;Vahl

M\(ya{Vc + MlWb + Ml) + Ml(Vb(Vc + M2) + M\VC)) (Va + Ml)(Vb + M\Wc + Ml)

ity, and range restriction effects The distribution of true validities used in this case is presented in Table 4 The mean and variance of this distribution were 50 and 031, respectively For this case, observed validity coefficients were produced by randomly selecting without replacement a true validity, criterion reliability, predictor reliability, and range restriction effect from the respective distributions m Tables 3 and 4 Once all values were used and 100 observed correlations had been generated, each of thefivevalidity generalization procedures was applied to obtain estimates of true validity mean and variance The process was repeated a total of nine times For each of the above cases, the fully corrected variance estimation equations for the five procedures (pre-

Table 3 Hypothetical Distributions of Criterion Reliability, Test Reliability, and Range Restriction Criterion reliability (a2)' 90 .85 80 .75 70 65 .60 .55 50 45 40 35 30

Test

/

reliability (b2f

/

SD of testc

u

/

3 4 6 8 10 12 14 12 10

90 85 80 75 70 60 50

15 30 25 20 4 4 2

10 00

1 00

701

70 65 60 56 52 47 41

5 11 16 18 18 16 11 5

8 6 4 3

*Ma = 7686, Va = 0093, Mai = 600, Va, = 0214 b Mt, = 8943, \b = 0022, Mh2 = 802, Vhi = 0066 C MU = 5945, Vu = 0146, Mui = 369, VU2 = 0285

6 49 6 03 5.59 5 15 4 68 4 11

387

VALIDITY GENERALIZATION Table 4 Hypothetical Distribution of True Unattenuated Unrestricted Validity Coefficients True validity

Frequency

94 90 86 82 78 74

2 3

70 66 62 58 54 50

5 6 7 8 9 10

1 1 1 2

procedures, range restriction effects were calculated differently in Callender and Osburn's independent multiplicative procedure as opposed to the procedures of Schmidt, Hunter, and their colleagues The equation for calculating c, for each particular study in the independent multiplicative procedure was c, = (u? + (1 - u:)M2r)"\

(4)

where each term is as previously denned The ecuation for calculating c, in the noninteractive and interactive procedures was c, =

- Mj)M;)"2 (Mi + («r -

(5)

where again, each term is as previously denned Equation 5 is equal to a formula previously given by Schmidt et al (1980) A proof of this identity is given in Appendix B Although Equations 4 and 5 are different mathematically, they yield approximately the same estimates for Mp and Vp We have chosen Equation 5 for the noninteractive and interactive procedures because it is much more consistent with the descriptions of Pearlman et al (1980) and Schmidt etal (1980) It should also be noted that the present investigation dealt with infinite sample sizes as did Callender and Osburn's study (1980) and, therefore, the sample size variance term (Ve) was set to zero in applying the validity generalization procedures

46 42 38 34 30 26 22 18 14 10 06

Results Case 1

sented in Tables 1 and 2) were applied to the observed validity distributions Similarly, the fully corrected mean estimation equations shown in Tables 1 and 2 were applied to these observed validity distributions Additional information concerning the noninteracttve, interactive, and independent multiplicative equations can be found in Schmidt et al (1980), Schmidt, Hunter, and Pearlman (1982), Callender and Osburn (1980), and Raju (Note 3) When applying each of the validity generalization

In Table 5, the Case 1 true validity estimates for the five procedures are presented. In general, all procedures were relatively accurate in estimating true validity; the differences in magnitude between the estimates were very small. TSA 2, however, yielded the most accurate estimates in almost all situa-

Table 5

Case 1 True Validity Estimates Procedure True validity 1 2

3 4 5 6 7 8 9

Noninteractive

Interactive

Independent multiplicative

TSA 1

TSA 2

1000 2000 3000 4005 5011 6023 7038 .8063 9100

1000 2000 3000 4005 5011 6023 7038 8063 9100

1000 1999 2997 3993 4988 5982 6977 7974 8980

0991 1982 .2972 3962 4952 5942 6935 7932 8937

1000 2000 2999 3998 4997 .5997 6999 8000 9019

Note TSA = Taylor Series Approximation

388

NAMBURY S RAJU AND MICHAEL J BURKE

Table 6 Case 1 True Validity Variance Estimates Procedure True validity 1 2 3 4 5 6 7

8 9

Noninteractive

Interactive

Independent multiplicative

TSA 1

TSA 2

-0004 -0017 -0038 -0064 -0094 -0123 -0148 -0160 -0148

0000 0000 0001 0002 0006 0013 0026 0049 0090

0000 0000 0002 0006 0014 0028 0052 0090 0148

0000 0001 0001 0001 -0001 -0004 -0008 -0013 -0018

0002 0006 0013 0021 0031 0042 0055 0069 0086

Note TSA = Taylor Series Approximation Actual Fp = 0000

tions. The procedures of Schmidt, Hunter, and their colleagues tended to slightly overestimate true validity, whereas the independent multiplicative procedure and TSA 1 tended to underestimate true validity As expected, the Mp estimates for both the noninteractive and interactive procedures were identical in all instances. The true validity variance estimates for the five validity generalization procedures are presented in Table 6 Table 6 indicates that TSA 1 was the most accurate estimator when true validity was greater than .2, whereas, the interactive procedure, independent multiplicative procedure and TSA 1 gave approximately the same estimates of Vp at the true validities of . 1 and .2 TSA 1 initially produced very slight overestimates up to the true validity of 4 and thereafter underestimated Vp. For each procedure, with the exception of TSA 1, the accuracy of the estimation generally decreased with increasing true validity. The noninteractive procedure produced negative estimates (or underestimates) for each of the nine levels of true validity. The interactive procedure, independent multiplicative procedure, and TSA 2 slightly but consistently overestimated true validity variance. Case 2 Table 7 shows that the five procedures, as expected, provided identical estimates of Mp in all instances. These identical estimates resulted from holding constant the artifactual effects Moreover, Table 7 illustrates that all

five procedures accurately estimated the mean true validity when there was no range restriction When range restriction did occur, each procedure overestimated Mp. The trend for the overestimates was to increase at both levels of range restriction (50% and 10% selected) as the criterion reliability and predictor reliability increased. When range restriction was set at 50% of those selected, the estimates of Mp for the five procedures were on the average .0239 larger than the actual Mp of .4950. At the prespecified selection ratio of .10, this difference was on the average .0347. The greatest overestimate of Mp occurred when the selection ratio was .10 and criterion and predictor reliabilities were held constant at 1.0. Similar to the estimates of Mp, all five validity generalization procedures accurately estimated true validity variance, as also shown m Table 7, when there was no range restriction In all situations involving range restriction, the five validity generalization procedures overestimated the actual true validity variance The true validity variance estimates of TSA 1 and TSA 2 were consistently more accurate than the estimates of Vp for the other three procedures m all situations. The Vp estimates yielded by TSA 1 and TSA 2 were the same when rounded to four decimals The estimates of Vp for TSA 1 and TSA 2 were on the average .0074 greater than the true validity variance of .0833. At the selection ratio of .50, this average difference was .0056. The average overestimation of Vp for these two procedures increased to .0096 at the se-

389

VALIDITY GENERALIZATION

lection ratio of .10. The noninteractive, interactive, and independent multiplicative procedures produced identical estimates of Vp. On the average, the estimates of Vp for these three procedures were .0331 larger than the actual true validity variance of .0833. More specifically, when range restriction was 50% and 10% selected, these procedures overestimated Vp by average amounts of .0249 and .0414, respectively.

dicates that all validity generalization procedures overestimated true validity Overall, TSA 1 yielded the most accurate estimates ofMp TSA 1 overestimated Mp by 0041 on the average. In addition, TSA 2 overestimated Mp by .0088 on the average. The nonmteractive and interactive procedures overestimated true validity by an amount of .0103 The independent multiplicative procedure overestimated Mp by an average value of .0078. The true validity variance estimates for Case 3 Case 3 are presented in Table 9. The nonThe true validity estimates for each of the interactive procedure produced, on the avnine trials are shown in Table 8. Table 8 in- erage, the most accurate estimates of Vp, folTable 7 Case 2 True Validity Mean and Variance Estimates \\ Range restriction (% selected)

Criterion reliability

Predictor reliability

Me All five procedures*

Noninteractive, interactive, and independent multiplicative procedures'"

V, TSA 1 and TSA 2C

100 100 100 100 100 100

1 0 10 10 6 6 6

1 0 8 5 1 0 8 5

4950 4950 4950 4950 4950 4950

0833 0833 0833 0833 0833 0833

0833 0833 0833 0833 0833 0833

100 100 100 50 50 50

3 3 3 1 0 10 1 0

1 0 8 5 10 8 .5

4950 4950 4950 5525 5366 5181

0833 0833 0833 1513 1281 1057

0833 0833 0833 .0982 0933 0884

50 50 50 50 50 50

6 6 6 3 3

1 0 8 5 10 8 5

5237 5173 5079 5079 5052 5012

1121 1046 0952 0952 0925 0889

0899 0882 0861 0861 0854 0846

10 10 10 10 10 10

10 10 10 6 6 6

10 8 5 10 8 5

5881 5567 5269 5355 5235 5124

2217 1581 1159 1266 1139 0997

1126 0997 0907 0930 0902 0871

10 10 10

3 3 3

10 .8 5

5124 .5086 5032

0997 0958 0908

0871 0862 0852

3

Note Actual Mp = 4950, V, = 0833 a The Mf estimates for the five validity generalization procedures were identical b The Vp estimates for the noninteractive, interactive, and independent multiplicative procedures were identical " The Vp estimates for TSA 1 and TSA 2 were identical

390

NAMBURY S RAJU AND MICHAEL J BURKE

Table 8

Case 3 True Validity Estimates Procedure Trial

Noninteractive

Interactive

Independent multiplicative

TSA 1

TSA 2

1 2 3 4 5 6 7 8 9

5102 5088 5116 5109 5135 5119 5077 5080 5097

5102 5088 5116 5109 5135 5119 5077 5080 5097

5077 5063 5091 5084 5110 5094 5052 5056 5073

5040 5027 5054 5048 5073 5058 5016 5019 .5036

5087 5073 5101 5094 5120 5104 5062 5066 5082

M

5103

5103

5078

5041

5088

Note TSA = Taylor Series Approximation Actual Mp = 5000

lowed by TSA 1 The noninteractive procedure underestimated true validity variance in four trials, whereas TSA 1 underestimated Vp in two trials The interactive procedure, independent multiplicative procedure, and TSA 2 slightly but consistently overestimated true validity variance. Discussion Estimates of True Validity The accuracy of the five validity generalization procedures in estimating Mp were tested in three cases In Case 1, the various procedures were quite accurate in estimating true validity, TSA 2 produced the most ac-

curate estimates The slight differences between the procedures of Schmidt, Hunter, and their colleagues and the independent multiplicative procedure can be traced to the manner in which the mean range restriction factor was calculated As previously noted, c was calculated differently in the noninteractive and interactive procedures than in the independent multiplicative procedure. In Case 2, all procedures overestimated Mp when there was range restriction The overestimates of Mp in the nomnteractive, interactive, and independent multiplicative procedures probably resulted from violation of the independence assumption (between p and c) when u was less than 1 0. (It should be

Table 9

Case 3 True Validity Variance Estimates Procedure Trial 1 2 3 4 5 6 7 8 9

M

Noninteractive 0548 0303 0337 0320 0358 0361 0265 0231 0307

Interactive 0650 0405 0440 0434 0462 0465 0352 0330 0410

Independent multiplicative 0621 0390 0423 0407 0444 0447 0354 0322 0395

TSA 1 0532 0329 0358 0344 0375 0378 0298 0270 0333

TSA 2

0337

0439

0423

0357

0397

Note TSA = Taylor Series Approximation Actual Vc = 0308

0575 0368 0398 0383 0415 0418 0336 0308 0372

VALIDITY GENERALIZATION

noted that in Case 2, a and b are uncorrelated with each other and with p and c, but c is uncorrelated with p only when u = 1 00.) When u was held constant at 1.0, c was fixed at 1.0 in each study. When u was held constant at .603 and 411, the value of c was less than 1.0 and, moreover, c and p were very highly (or almost perfectly) correlated In such instances, Callender and Osburn (1980) have noted that the use of ML to correct Mr upward results in an overestimation of Mp. Although the assumption of pairwise independence of p, a, b, and u was satisfied in Case 2 because the correction factor for TSA 1 and TSA 2 was the same as that for the other procedures, the TSA 1 and TSA 2 procedures also overestimated Mp by the same amount All procedures overestimated Mp in Case 3 TSA 1 consistently provided the most nearly accurate Mp estimates in this case, followed by the independent multiplicative procedure, TSA 2, and the identical estimates of the nonmteractive and interactive procedures Again, the differences between the Mp estimates of the Schmidt-Hunter procedures and those of the independent multiplicative procedure could be traced to the manner in which c was calculated. Overall, TSA 1 and TSA 2 gave slightly better estimates of Mp, especially in Cases 1 and 3, where the Mp estimates from the five procedures differed Because the differences between the estimates from the five procedures are very small, the practical significance of the superiority of TSA 1 and TSA 2 estimates is minimal.

391

terms in the true validity variance estimation equation Raju (Note 3) showed that under the assumption that a, b, and c are independent and variable, the numerator of the equation for Vp should contain the terms (Vb + M\) and (Vc + Ml) The nonmteractive procedure does not include these terms Instead, it assumes that each is equal to 1. Callender and Osburn (1980) have shown that when this assumption is incorrect, it results in an increase in the artifactual (predicted) variance The net result in this study was for the sum of the negative variance terms (artifactual variance) to be greater than the observed validity variance, Vr. Consequently, negative estimates of residual variance and Vp were made As expected, in Case 2 all procedures produced accurate estimates of Vp when there was no range restriction When range restriction was introduced, the identical Vp estimates of TSA 1 and TSA 2 were, m many situations, considerably more accurate than the identical estimates of the other three procedures The fact that the nonmteractive, interactive, and independent multiplicative procedures did poorly m estimating Vp may have been due to the high correlation between c and p. As previously indicated, c and p are almost perfectly correlated in Case 2 when u is less than 1 0 Because the noninteractive, interactive, and independent multiplicative procedures do not take this dependency into account (in fact, these procedures assume that the correlation between c and p is 0), the use of Mc as part of the correction factor will overestimate Vp (Callender and Osburn, 1980). In the case of TSA 1 and TSA 2, the assumption of pairwise independence Estimates of True Validity Variance of p, a, b and u is fully met in Case 2, which The accuracy of the validity generalization helps to explain the improved degree of acprocedures in estimating Vp was also tested curacy of Vp estimates obtained with these m three cases. In Case 1, where p was con- procedures. Yet the fulfilling of the assumpstant and all artifacts were vaned, TSA 1 tion of pairwise independence of p, a, b, and yielded the most accurate estimates in almost u does not appear to be enough in accurately all instances The Vp values in Case 1 for the estimating Mp in Case 2. interactive procedure, independent multipliIn Case 3, where all factors were vaned, cative procedure and TSA 2 overestimated all procedures overestimated Vp. On the avVp by a small amount in most cases In con- erage, the nonmteractive procedure yielded trast, the nonmteractive procedure underes- the most accurate estimate of Vp, followed timated Vp at all levels of true validity in Case by TSA 1. TSA 2 produced the third most 1. A plausible reason for these underesti- accurate estimates of Vp, followed by the inmates can be traced to the omission of certain dependent multiplicative and interactive pro-

Page Missing

Page Missing

394

NAMBURY S RAJU AND MICHAEL J BURKE Appendix A

This appendix contains the proofs of Taylor Series Approximation 1 (TSA 1) and TSA 2 for assessing validity generalization with the correlation model

pK

"

Jrlp

JrJpJVJ u

Similarly, we can show that

TSA 1

Vr = V(P) + Ve,

2 \Mb2

(Al)

where P = n(o2V/2(h2V/2 P(a)

(b}

(\ + (u2 -

-

= B,

/>3(ij)(l

p, a2, b2, and u are pairwise independent Let us rewrite Equation 3 as

F

(A4)

MM2

M

^j = C, (A6)

Mh2Ml

- P3(v) = A Mu \)P2a2b2y2

and is a function of p, a2, b2, and u Let Mp. Ma2, Mh2, and Mu denote the means of p, a2, b2, and u, respectively, and let y be a vector consisting of Mp, Ma2, Mb2, and Mu. Furthermore, let the partial derivatives of P be denoted as follows.

(A5)

-Ml)

(A7)

where P(v) =

(1 + (Mi -

\)M2pMaiMb2){'2

(A8)

It should be noted that A, B, C, and D depend upon the means of the distributions of p, a2, b2, and u Now, from Equations A3 to A7, the variance of P can be written as VP = A2VP + B2Va2 + C2Vb2 + D2VU

2

db '

du

From Equation Al, we have

Then, using the Taylor series expansion, the variance of P can be written as

VP=

2

V = V + V = A2VP + B2Va2 + C2Vb2 + D2VU + Ve

P'HvW,

Solving this equation for Vp, we obtain + (terms of order K~2 or less), 2

(A2)

2

where i, j = p, a , b , and u (Kendall & Stuart, 1977, p. 247), and k is the number of validity studies It should be noted that P'(y) refers to the value of F at Mp, Ma2, Mbi, and Mu If we now asume that p, a2, b2, and u are pairwise independent and ignore the terms of order k~2 or less, then the variance of P can be written as

Vp =

2

2

(A3)

2

Since r = P + e, the mean of r can be expressed as Mr = Mp + Me and, according to Kendall and Stuart (1977, p 246), M P = />(,) =

\)MJMa2Mb2y/2

(1 + (Ml -

Since Me is generally assumed to be zero, Mr = Mp = P(v) Solving this equation for Mp, we obtain

p,a ,b ,u

To simplify this expression further, let us consider the specific expressions for the four partial derivatives of P First, J

+(M2_P

- u2) pU2

(A 10)

TSA 2

pa2b2u +(u \)P2a2b2)u2

p, a, b, and u are pairwise independent Let us rewrite Equation 3 as

2

o

r2 + M?(i - M J ) ) ) 1 / 2

l)p2a2b2u2)V2

Kr = V(Q) + Ve, where = Pab

u ( 1 + (w2 -

\)p2a2b2y2

(All)

395

VALIDITY GENERALIZATION and is a function of p, a, b, and u Following the mechanics of TSA 1, the partial denvatives of Q are

timate of the vanance of Q can be written as VQ = E2VP + F2Va + G2Vh + H2VU From (Al), we have

= £, (A12) = E2l-P + F2Va + G2Vb + H2VU + V,

(A 13)

Solving this equation for l'p, we obtain (A 14) and (A15)

rr=V,-V.-F*V.-G>Vb-H%

Since r = Q + e, the mean of r can be expressed as Mr = MQ + Me and, according to Kendall and Stuart (1977, p 246),

where MpMaMbMu

( A n )

MQ = 0(0) =

MpMaMhMu

l*Zh = Mr (A16)

Note that £, F, G, and / / depend upon the means of the distributions of p, a, b, and u, denoted here by a vector 6 = (Mp, Ma, Mb, Mu) Now, using the Taylor Series expansions given in TSA 1, an es-

Since Mc is generally assumed to be zero, Mr = MQ = Q(0) Solving for Mp, we obtain M.=

Appendix B According to Equations 1 and 2, c, =

1/2

(1 + ( u f -

If we let R2 = pfr^ryy, the above equation can be rewritten as c, =

u,

(Bl)

our Equation 5, for estimating c, We have chosen the former estimation method for the nomnteractive and interactive procedures because it appears to be more consistent with the detailed computational procedures given by Pearlman, Schmidt, and Hunter (1980, pp 402-403, Section C) The formula for estimating R can be written, according to Lord and Novick (1968), as (B2)

which is identical to an equation given by Schmidt, Gast-Rosenberg, and Hunter (1980, p 658) It should be noted that R, is the true validity attenuated for criterion and predictor reliabilities Since R, is not generally known, we must estimate R, with known quantities in order to compute c, One way to estimate R, is to first replace it with mean true validity (R) and then estimate R with known quantities. Another method for estimating R, is to rewrite R, as Callender and Osburn (1980) have done and use their procedure, which is given as

where Mu and Mr are the means of range restrictions («,) and observed correlations (r,), respectively If R in Equation B2 is substituted for R, in Equation Bl, we will obtain Equation 5

Received September 20, 1982 Revision received March 28. 1983