probability and utility assumptions underlying use of ...

Journal of Applied Psychology 1974, Vol. 59, No, 4, 456-464

PROBABILITY AND UTILITY ASSUMPTIONS UNDERLYING USE OF THE STRONG VOCATIONAL INTEREST BLANK FRANK L. SCHMIDT 1 Michigan State University Decision implications of Strong's assumption of equal base rates for his menin-gcncral (MIG) and individual occupational groups are explored. Using the Strong Vocational Interest Blank (SVIB) Physician's scale as an example, it is demonstrated that when realistic base rates are assumed, total decision error rate using the SVIB—even when optimal cutting scores are employed— is at best about equal to that resulting from prediction using base rates alone. It is shown that use of the SVIB as recommended by Strong implicitly assumes that false negative errors are much more serious than are false positives. Finally, it is shown that when "reasonable" utility or gain matrices are assumed, use of the SVIB can reduce total error rate despite the limitations imposed by low occupational base rates, and that this effect is greater when statistically optimal cutting scores, rather than those recommended by Strong, are used.

In many practical decision-making tasks faced by psychologists, scores on some continuous test or variable are used to make dichotomous decisions. In a study that is now a classic, Meehl and Rosen (1955) demonstrated that cutting scores used in such decisions must take into account population base rates, that is, the relative frequency of the category or group of interest. They showed that a device which usefully reduces decision or classification errors when the base rate of the dichotomous phenomenon of interest (e.g., schizophrenic versus normal or shoplifter versus honest shopper) is near .50 may be useless—or may even lead to increased errors—when the base rate is very high or very low. They presented conditions that must be met for the test to achieve a greater degree of accuracy than would result from use of the base rates alone. Dawes (1962) later provided a more easily understood analysis in terms of conditional probabilities. Rorer, Hoffman, LaForge, and Hsieh (1966) presented an analytic (as opposed to empirical) solution to the determination of optimal cutting scores, and Rorer, Hoffman, and Hsieh (1964) provided extensive tables of optimum cutting scores for groups differing both in 1 Requests for reprints should be sent to Frank L. Schmidt, Department of Psychology, Michigan State University, East Lansing, Michigan 48824. The author is grateful to John Hunter and Robert Forsythe for helpful comments on an earlier draft of this article.

base rate and variance. These tables assume that all errors are equally costly and all correct classifications are equally beneficial; a subsequent study (Rorer, Hoffman, & Hsieh, 1966) presented a method of determining optimal cutting scores when false positives and negatives are unequal in cost and true positives and negatives are unequal in benefit. The decision theory principles explored in these studies can be applied to an evaluation of the Strong Vocational Interest Blank (SVIB)—perhaps the most commonly used instrument in occupational counseling. The purpose of the present study is to show that application of these principles leads to conclusions about the usefulness of the SVIB that are quite different from those reached by Strong (1955), Strong and Campbell (1966), and Campbell (1968, 1971). The following analysis applies equally to the 1966 revision and to the earlier form. Similarly, while it is in terms of the men's form, the analysis generalizes easily in all its essentials to the women's form of the SVIB. IMPACT OF BASE RATES ON EXPECTANCIES AND ON TOTAL ERROR RATES Meehl and Rosen (1955) and Dawes (1962) have pointed out that criterion validation all too often results in a statement of test validity in terms of P(Rj/Gi), that is the probability of Response j for subjects in Group i (e.g., the probability that schizo-

456

PROBABILITY AND UTILITY ASSUMPTIONS UNDERLYING USE OF SVIB phrenics will score above 70 on a test). The desired index of predictive efficiency, however, is P(Gi/Rj), the probability of group membership, given the response (e.g., the probability that the respondent is a schizophrenic given that he scores above 70). These two probabilities are related by means of Bayes' theorem: P(Ri/Gi)P(Gi) P(R,) '

P(Gs/R,) =

Strong (1955) successfully avoided this common error and presented validity evidence for the SVIB in the form P(G i /R j ), where GI is the occupation in question and Rj is a letter grade on the SVIB scale for the occupation in question. In summarizing the results of his 18-year follow-up study of 884 Stanford University students, Strong (1955, p. 43) presented the probabilities (expectancies) shown in Figure 1 as averages across 16 occupations. The probability of being in a given occupation given that one obtained an A—, for example, on the scale for that occupation 18 years earlier is .74. These probabilities are quite impressive, and this summary is widely quoted as evidence of the SVIB's effectiveness (e.g., Anastasi, 1968, p. 472; Super & Crites, 1962, p. 440). But, as pointed out by Dolliver (1969), such evidence is both erroneous and highly misleading. Each SVIB scale is designed to discriminate maximally between that occupation and the men-in-general (MIG) group, a large heterogenous group of men in various professional and paraprofessional occupations. Because base rates in Strong's (1955, pp. 40-41) MIG Letter Rot ing Score

A*

55 - 70

A

45-54

B+

40-44

B

35-39

8

30-34

C

Below 30

,8B 174 162 149

7

|36

zui

0

10 20 30 40 SO 60 7O 80 9O 100

Chances in 100 of Employment in Occupation FIGURE 1. Chances in 100 of employment in occupation given specific letter grade on the Strong Vocational Interest Blank 18 years earlier (Strong, 1955, p. 43).

10

20

30

40

SO 60

457

70

80

FIGURE 2. Distributions of the men-in-general and specific occupation groups under the assumption of equal base rates.

group were different for different occupations and because Strong felt that within occupations, these base rates would vary as between universities and geographical regions, he decided to express expectancy ratios for all scales "in terms of samples of equal size." The implications of this procedure can be seen by comparing Figures 2 and 3.2 In comparing each occupation with the MIG group, Strong assumed the situation shown in Figure 2, that is, an occupational group equal in size to the MIG group. This assumption is obviously highly unrealistic; none of the occupations scaled on the SVIB even approximates the relative frequency of all other occupations combined—and this holds true whether we consider base rates in Strong's MIG group or in the United States male employed professional-technical population. Figure 3 illustrates a more reasonable assumption; it depicts the base rate for the occupation as being much smaller than that of the nonoccupation, or MIG group. Operating on the basis of this assumption leads one to estimates of P(Gi/Rj) much smaller than those shown in Figure 1, with the degree of reduction in size depending on the estimate chosen of the base rate of the occupation. As an example, consider the physician's scale. Physicians constituted 6% of the original MIG group (Strong, 1955, p. 68); they make up 5.1% of the United States employed male professional population (U.S. Bureau of the Census, 1963).8 When considered in relation 2 Scaling of the abscissa in Figures 2 and 3 approximates that of the typical SVIB scale (i.e., XUIB = 30, Xoccup. si 50, and SDum s* SDaccu\>. = 10). 3 1960 census figures; at the time of this writing, national 1970 census breakdowns by occupational group were not yet available. But it is unlikely that the 1970 base rates will be significantly different from 1960 figures.

FRANK L. SCHMIDT

458

TABLE 1 P(Gi/Rj) FOR THE STRONG VOCATIONAL INTEREST BLANK PHYSICIAN'S SCALE UNDER FOUR BASE RATE ASSUMPTIONS

-10

10

20

30

40

SO

60

70

80

90

FIGURE 3. Distributions of the men-in-general and specific occupation groups under the assumption of unequal base rates.

to the combined U.S. employed male professional, managerial, sales, and farm populations—all of which occupational categories are included in the SVIB MIG group—the physician's base rate is only 1.6% (U.S. Bureau of the Census, 1963). In certain population subgroups, of course, base rates may be higher. For example, in the population of undergraduate biology majors the proportion of future physicians may be as high as .20. Assuming a 20-point mean difference between physicians and the MIG group (Strong, 19SS, p. 37) and the standard deviations of 13 and 10 for the MIG and physician's groups, respectively (Strong, 19SS, p. 68), application of Bayes' theorem 4 yields the expectancies shown in Table 1 for the base rates of .SO and .20, and the rounded base rates of .05 and .02. (The base rate .060 was excluded because of its closeness to .051.) The physician's scale discriminates somewhat better than the average of all SVIB scales, and the result is a very impressive set of expectancies under the assumption of equal base rates. For example, a college student scoring A— is given 8 chances out of 10 of being a physician 18 years later. However, these expectancies drop off sharply when more realistic base rates are entered into the equa4 Values inserted into the right side of Bayes' equation were calculated as follows: /"(Rj/Gi) was determined for the physician's distribution for each letter grade using normal curve tables. P(Gi) is, of course, the assumed physician's base rate. F(Rj) = P(R,/Gt) - P ( G i ) plus the probability of the response in the MIG group times the assumed base rate of the MIG group, that is, l - P ( G i ) . These calculations, and those for Table 2, assume normality of both MIG and Occupation scale distributions. Although there are some apparent departures from normality in some occupational scales, they are not large enough to significantly affect the figures in Tables 1 and 2.

Base rate

Letter rating

A+ AB+

B BC

Score

55-70 45-54 40-44 35-39 30-34 Below 30

.50

.20

.05

.02

.92 .80 .60 .41 .21 .04

.74 .50 .28 .15 .07 .01

.37 .16 .07 .04 .02

.19 .07 .03 .02 .01

.002

.001

Note. Essentially, P(Gi/Rj) is the probability of a man's being a physician, given that he obtained a certain score on the physician's scale 18 years earlier.

tion. When the base rate is taken as .20, a student scoring A— has even chances of not becoming a physician. When .05, rather than the .02 which is perhaps more realistic in most situations, is employed as the base rate, none of the expectancies is as large as .50. No matter how high the student scores, the probability is always much higher that he will become a nonphysician than a physician. This problem is not peculiar to the occupation of physician. In fact, most of the occupations on the SVIB that could be found in the census figures showed smaller base rates than the physician category. Another approach to this problem is via an examination of the total error rate—the percent of all decisions that are in error—as TABLE 2 PERCENTAGE OF ALL DECISIONS INCORRECT FOR STRONG VOCATIONAL INTEREST BLANK PHYSICIAN'S SCALE AS FUNCTION CUTTING SCORE AND BASE RATE Base rate Cutting score

45 (A.-) 40 (B+) 40.13" 49.5h 65.39=

.50

.20

21.7 19.0 18.5

16.2 20.8 — 15.7

— —

—

.05

.02d

13.4

12.9 21.9 — — —

21.2

— — 4.9

" Optimal cutting score for base rate = .50. i> Optimal cutting score for base rate = .20. c Optimal cutting score for base rate = .05. 11 For base rate = .02, there is no cutting score that results in fewer errors than prediction using base rate alone.

PROBABILITY AND UTILITY ASSUMPTIONS UNDERLYING USE OF SVIB a function of cutting score and assumed base rate. Strong (19SS, p. 54) has advocated that students "seriously consider" an occupation on which they score B+ or above, thus advising counselees to tentatively classify themselves into such an occupation rather than into the MIG group. The more stringent cutting score of A— is also sometimes used. In addition, the statistically optimal cutting score can be determined for each assumed base rate. Table 2 presents total error rates for these cutting scores as a function of four base rates.5 When a base rate of .50 is assumed, Strong's recommended cutting score of B+ is close to optimal; in addition, all cutting scores reduce the error rate considerably below that resulting from use of the base rate alone (50%). When the base rate is taken as .20, A— is close to optimal as a cutting score, while the B+ suggested by Strong produces an error rate above the 2Q% base-rate level. When the base rate is taken as .05, both letter grade cutting scores produce error rates considerably above the 5% level that would result from use of the base rate alone. Even the optimal cutting score reduces base-rate error by only .1%. When .02 is taken as the base rate, no cutting score can be drawn which produces fewer errors than does the base-rate prediction that no one will become a physician (2% error rate). Even if we are willing to assume a .05 base rate for future physicians in most counselee populations, the reduction of .1% in total error hardly makes use of the SVIB appear worthwhile. Clearly, if use of the Strong can be justified, justification must be on the basis of inequality (i.e., differential weighting) of classification errors. 5

For the physician's scale, X = 50, SD = 10; for the MIG distribution, X = 30, SD = 13 (Strong, 1955, p. 68). For each cutting score, total error rate is percent of false positives in the MIG group (determined from normal curve tables) times the assumed MIG base rate plus the percent of false negatives in the physician's group (also from normal curve tables) times the assumed physician base rate. Optimal cutting scores for all base rates except .20 were calculated by interpolation using the Rorer, Hoffman, and Hsieh (1964) tables. Since the base rate of .20 is not contained in these tables, this optimal cutting score was calculated using Equation 1 of Rorer, Hoffman, LaForge, and Hsieh (1966).

459

INEQUALITY OF ERRORS The SVIB is probably most realistically viewed as one step in a sequential career decision-making process. The individual counselee will typically obtain C or B— ratings on most of the 54 occupational scales, and most of these occupations are usually then tentatively eliminated from further consideration. The occupation that is in fact optimal for the counselee will probably be found among the relatively small number of scales on which he scores A or B+, and these occupations are thus retained for closer scrutiny. If we assume that the individual can then eliminate the inappropriate occupations (scales) in this group by some other means (e.g., financial or practical considerations, personal insight or preference, consideration of intellectual ability requirements, etc.), then false positives become less important than false negatives. The cost of false negatives is higher because, once assigned to the MIG group on a given scale, the counselee does not ordinarily investigate that occupation further, thus making the error a potentially permanent one. False positive errors, on the other hand, are usually remedied when the individual more closely scrutinizes the occupations on which he obtains an A or B + rating. Given that the two different types of error are unequal in cost, the next question is whether correct classifications, true positives and negatives, are equal or unequal in value. Practical considerations, for example, the fact that for each individual there are many true negatives but only one true positive; the fact that many true negatives can be correctly arrived at by the individual without benefit of the SVIB, etc., seem to indicate that true positives are of greater value than true negatives. Nevertheless, it might be argued that there are three sets of assumptions about true positives and negatives that are plausible. Assumption 1 states that gain in utility from a true positive (G ++ ) and gain from a true negative (G__) are both zero; true positives and negatives merely function (and function equally) to avoid losses in utility that result from false positives and negatives. Assumption 2 holds that true positives and negatives produce equal positive gains. Assumption 3

460

FRANK L. SCHMIDT

assumes that both gains in utility are positive, but that the value of a true positive exceeds that of a true negative. These three assumptions can be expressed algebraically as follows: 1. G++ = G__ = 0 2. C++ = G__ > 0 3. G++ > G__ > 0. Sets of assumptions other than those three (e.g., G++ < G__ and G++ = G__ < 0) can be eliminated as implausible. The decision principles presented by Rorer, Hoffman, and Hsieh (1966) can now be used to shed light on the extent of inequality that is implicitly being assumed between false positive and false negative errors under each of these three sets of assumptions when the SVIB is used in counseling as recommended by Strong (19SS), Strong and Campbell (1966), and Campbell (1968, 1971). That is, we can recover some of the properties of the gain or utility matrix implicitly assumed by Strong (19SS). Given a gain or utility matrix such as that shown in Table 3, Rorer, Hoffman, and Hsieh (1966) showed that B G++ — G + _ _ B'

I-B'G__-G_+

i - B''

LJ

where B = the true base rate for the occupational group and B' = the base rate adjusted for the inequalities in the gain or utility matrix. B' is the base rate needed to enter the table of optimal cutting scores, given a particular B and utility or gain matrix. For convenience let G

- G_ +

TABLE 3

Predicted

Men-in-general ( — ) Occupation (+) Note. G = gain.

G =

B'(l - B ) (1 - B')B'

[2]

In computing all expectancy ratios on the basis of equal base rates, Strong was in effect assuming B' = .SO. Using this value for B', Equation 2 yields G = 4 when B = .20, G = 19 when B = .05, and G = 49 when B is taken as .02. Then under Assumption 1 that G__ = G++ = 0,

G+G-+ G+.. _ 5= ° ^7~

G.20

and

=

C-

4 1' 19 1'

49

where .20, .05, and .02 refer to base rates. Thus, if true positives and negatives are considered to each represent zero gain, then use of the SVIB physician's scale assumes that a false negative is 4, 19, and 49 times as harmful as a false positive error if the true base rate is taken as .20, .05, and .02, respectively. (The reader should recall that both G+_ and G^+ are negative numbers.) If we assume G __ = G++ > 0, then when the base rate is .20, .

~

~ = A4.

~ VJ __ — (j __ (_

Substituting G++ for G__ and solving, we get

= G.

GAINS MATRIX FOR AN INDIVIDUAL OCCUPATIONAL SCALE ON THE STRONG VOCATIONAL INTEREST BLANK

Actual

Then, solving for G in Equation 1 gives

Men-ingeneral (-)

Occupation ( + )

G__ G+_

G_ + G+ +

Since G_+ is a negative number (a negative "gain"), as G++ and G__ increase a false negative comes to be considered more than four times worse than a false positive. Similarly, when the base rate is taken as .05,

And, finally, when the base rate is .02, we get TT^ = 49 -

PROBABILITY AND UTILITY ASSUMPTIONS UNDERLYING USE OF SVIB As G+4. and G__ increase, false negatives become more costly relative to false positives. Thus, if we consider true positives and negatives to have positive and equal value, use of the SVIB physician's scale as advocated by Strong (19SS) assumes that false negatives are more than four times as serious as false positives when the base rate is taken as .20. When the base rate is taken as .OS, use of the SVIB assumes that false negatives are more than 19 times worse than false positives. When the base rate of .02 is used— the most realistic of the three base rates for most high school and college counselee populations— the implicit assumption is that classifying a counselee into the MIG group when he will in fact enter the occupation is more than 49 times as serious as the opposite error. Finally, if we assume (Assumption 3) that G+t > G__ > 0, then, when the base rate is taken as .20, G++ - G+_

Solving for G^/G_+, we get G+- _ G_+

+

G ++ - 4G— G_+ '

Since G_+ is a negative number, G+_/G_+ will be less than 4 only when G++ > 4 G— For all other permissible values of G l + , a false negative will be considered more than four times as serious as a false positive. When the base rate is taken as .05, ~ = 19 +

G_+

- 19G__ vj-+

Here the value of a true positive (G t+ ) must be considered to be more than 19 times the value of a true negative (G__) for the cost of a false negative (G+_) to be considered less than 19 times as great as a false positive (G_ + ). When the base rate is .02, G-,

G-+

With this, the most realistic of the base rates, the cost of a false negative (G + _) is considered greater than 49 times the cost of a false positive (G_J unless the value of a

461

true positive (G ++ ) is considered to be equal to or greater than 49 times the value of a true negative (G__). On logical and rational grounds, both Assumption 3 and a base rate of .02 appear the most realistic for most situations in which the SVIB is used. On the same grounds, it would appear unlikely that true positives are in fact considered more than 49 times as useful as true negatives. The conclusion indicated, then, is that use of the SVIB physician's scale as recommended by Strong (1955, p. 54) implicitly assumes that false negative errors are approximately 49 times as harmful as false positive errors.6 ACCURACY GIVEN A "REASONABLE" GAINS MATRIX Strong's (1955) assumption that the adjusted base rate (B') is .50 leads backward to implicit gains matrices with G = 4, 19, and 49 when the true base rate (B) is taken as .20, .05, and .02, respectively. Ordinarily, of course, the process is reversed: The four values in the gains matrix are specified, and these values, along with B, are entered into Equation 1 to solve for B'. B' is then used to enter the Rorer, Hoffman, and Hsieh (1964) tables to find the optimum cutting score and the adjusted total error rate. An important question then, is whether, given gains matrices containing "reasonable" values, use of the SVIB leads to an adjusted total error rate lower than B', the error rate obtaining when prediction is on the basis of B' alone. The 0 Some counselees may expect the SVIB to indicate not the specific occupation that would be optimal for them but rather a cluster of family of occupations, most of which would in practice be suitable from the viewpoint of interests. When the SVIB is used this way, the appropriate unit of analysis would be the 11 occupational families on the SVIB. With this kind of analysis, the base-rate problem would be reduced (few families would have base rates as low as .02), and the number of decision errors, especially false positives, would be reduced. While this may be a realistic model for the use of the SVIB for many counselees, especially in view of the high occupational scale intercorrelations within most occupational families, it does not seem to be that of Strong (1955). Strong's focus is on the individual occupations. Nevertheless, a subsequent study is planned using the occupational families as the unit of analysis.

FRANK L. SCHMIDT

462 TABLE 4

A "REASONABLE" GAINS MATRIX TOR INDIVIDUAL STRONG VOCATIONAL INTEREST BLANK SCALES Predicted Actual

Men-in-general ( — ) Occupation (+)

Men-ingeneral ( — )

Occupation (+)

+1 -10

+ 10

-1

gains matrix shown in Table 4 might be reasonable for many counselees. Table 4 assumes that correct identification of the individual's future occupation (Cell + + ) is 10 times as useful as the correct identification of the occupation as inappropriate for the individual (Cell ). Similarly, incorrect rejection of the occupation as inappropriate (Cell H—) is 10 times as harmful as incorrectly labeling the occupation as appropriate (Cell —+ ).r In an individual decision-making process of this sort, actual gains matrices would, of course, vary from counselee to counselee, depending on individual differences in value systems (Cronbach & Gleser, 196S, pp. 7-9). For other counselees, a "reasonable" gains matrix might be produced by substituting +20 for the +10 in cell G++ of Table 4 and a —20 for the —10 in cell G+_. Such counselees would consider false negatives more costly and true positives of greater value than individuals characterized by the gains matrix in Table 4. This we refer to as the second or alternative "reasonable" gains matrix. For both these "reasonable" gains matrices, Table S shows for each of three base rates: (a) B' (from Equation 2), (b) total adjusted error rate,8 and (c) percent reduction in error using (a) optimal cutting scores (from Equa7

These statements, and the analyses of the previous section, assume measurement of the utilities on a ratio scale. However, as pointed out later, empirical estimation of the utility of the SVIB for a given individual requires measurement of the four utilities in his gains matrix only on an interval scale. As can be seen in Equation 1, this follows from the fact that it is the ratio of differences rather than the direct ratio of utilities that is used in computing B'. 8 Both total adjusted error and percent reduction in error are, of course, averages that would be expected over a large number of individuals with iden-

tion 1 of Rorer, Hoffman, & Hsieh, 1966) and (b) Strong's (19SS, p. 54) suggested cutting score of 40 (B + ), When the base rate is taken as .20, use of the SVIB reduces total adjusted error rate below levels that would result from prediction using B' only, but these gains are considerably larger when optimum cutting scores, rather than Strong's suggested score, are employed. This difference is especially large for the alternative gains matrix. In this instance the reduction in total error that would result using Strong's cutting score is so small (.5%) as to be negligible. When the base rate is taken as .05, the cutting score advocated by Strong leads to a significant reduction in total error for the first gains matrix and is only marginally less efficient than the optimal cutting score. For those counselees whose values correspond to the alternative gains matrix, Strong's cutting score is essentially identical to the optimal cutting score at this base rate. Under this set of circumstances, procedures recommended by Strong (1955, p. 54) for the use of the SVIB are ideal. The optimal cutting score, when B = .02, works out to 51 for the first gains matrix and produces a total adjusted error rate of 13.2% (versus the 17% expected when predicting using adjusted base rate alone). This constitutes a 22.4% reduction in error. However, when the cutting score of 40 recommended by Strong is used instead of the optimum cutting score of 51, total adjusted error rate is 21.0%—a greater error rate than that which obtains when predictions are made tical gains matrices. The model used in this study— which is general statistical decision theory with differential utilities—was developed for situations in which many decisions are made under similar circumstances. In the case of individual decisions, each individual's gains matrix is potentially unique (though it is likely in practice that some individuals will have essentially identical matrices). When applied as above, the model implicitly assumes, for each gains matrix, a hypothetical population of individuals all with the same gains matrix; it then gives the average error that would obtain over this population. For any individual with this gains matrix, this is the best estimate of the value of the SVIB for him. This estimate must be based on a model which assumes a number of similar decisions; each individual's decision is a unique event which never recurs and thus cannot be modeled in this sense.

PROBABILITY AND UTILITY ASSUMPTIONS UNDERLYING USE OF SVIB

463

TABLE 5 B', TOTAL ADJUSTED ERROR RATE, AND PERCENT REDUCTION IN ERROR TOR Two "REASONABLE" GAINS MATRICES AT THREE BASE RATES (B) Optimum cutting score

B

B'

Cutting score

Total adjusted error rate

Strong's (19SS) cutting score

% reduction in error

Cutting score

Total adjusted error rate

% reduction in error

17.7 20.2 21.0

39.0 40.5 -23.5

16.9 18.9 20.3

61.4 30.0

"Reasonable" gains matrix in Table 4 (G = 10)

.20 .05 .02

.71"

.34 .17

34.4 44,0 51.0

15.0 18.6 13.2

48.3 45.3 22.4

40 40 40

Alternative "reasonable" gains matrix (G = 20)

.20 .05 .02 a

.83" .51"

.29

30.8 40.4 45.7

10.4 18.9 17.7

38.8 61.4 39.0

40 40 40

.5

Since B' > .50, error rate for base-rate prediction is 1 — B'.

from the adjusted base rate alone. On the CONCLUSION other hand, when the alternative gains matrix In summary, the SVIB, if carefully used, is assumed, use of Strong's cutting score leads probably has a significant positive utility for to a useful reduction in total decision error, most individuals, despite the limitations on though not as great a reduction as is pro- predictive efficiency imposed by low occupaduced by the optimal cutting score. Of the tional base rates. This utility, however, can three base rates examined in the present conbe cancelled or even rendered negative under text, .02 is probably the most realistic for most of the high school and college counselee some circumstances if nonoptimal cutting populations with which the SVIB is used. scores—including those recommended by Utility of this instrument when used with Strong—are employed. Whether or not this Strong's suggested cutting score in these pop- latter effect occurs depends directly on the ulations will be negative for those individuals relative negative values placed on false posiwhose subjective gains matrix corresponds to tives and negatives and the relative positive the matrix in Table 4. As the counselee's values placed on true positives and negagains matrix moves away from the Table 4 tives, that is, on the gains matrix of the indimatrix and toward our alternative gains vidual. These conclusions point to a need for matrix, the positive utility of the SVIB for the measurement of actual counselee gains him increases. matrices.9 If gains matrices of individual In conclusion, when the utility of the SVIB is calculated using more realistic base counselees could be assessed, optimal cutting rates, the instrument still shows a positive scores and expected utility of the SVIB could utility in many situations, given a "reason- be assessed for each individual. 9 able" gains matrix. Utilities are highest, Measurement problems in the present context however, when optimal cutting scores rather are not as complex as they might seem at first than those recommended by Strong are used, glance. The four utilities in an individual's gains and under certain, possibly common, combi- matrix need to be measured on only an interval A ratio scale, that is, a scale with a rational nations of base rate and gains matrix, use of scale. zero point, is not necessary (see footnote 6). StanStrong's cutting score can lead to greater dard psychometric procedures are generally considerror rates than are produced by prediction ered to give adequate approximations to interval scales (Nunnally, 1967, chap. 1). from base rates alone.

FRANK L. SCHMIDT

464 REFERENCES

Anastasi, A. Psychological testing. New York: Macmillan, 1968. Campbell, D. P. The Strong Vocational Interest Blank: 1927-1967. In P. McReynolds (Ed.), Advances in psychological assessment. Vol. 1. Palo Alto, Calif.: Science & Behavior Books, 1968. Campbell, D. P. Handbook for the SVIB. Stanford, Calif.: Stanford University Press, 1971. Cronbach, L. J., & Gleser, G. C. Psychological tests and personnel decisions. Urbana: University of Illinois Press, 196S. Dawes, R. M. A note on base rates and psychometric efficiency. Journal of Consulting Psychology, 1962, 26, 422-424. Doffiver, R. H. "3.5 to 1" on the Strong Vocational Interest Blank as a pseudo-event. Journal of Counseling Psychology, 1969, 16, 172-174. Meehl, P. E., & Rosen, A. Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. Psychological Bulletin, 1955, 52, 194-216. Nunnally, J. C. Psychometric theory. New York: McGraw-Hill, 1967. Rorer, L. G., Hoffman, P. J., & Hsieh, Kuo-Cheng. Tables of optimum cutting scores to discriminate

groups of unequal size and variance. Eugene, Oregon: Oregon Research Institute Technical Report, 1964, 4, (3). Rorer, L. G., Hoffman, P. J., & Hsieh, Kuo-Cheng. Utilities as base rate multipliers in the determination of optimum cutting scores for the discrimination of groups of unequal size and variance. Journal of Applied Psychology, 1966, SO, 364-368. Rorer, L. G., Hoffman, P. J., La Forge, G. E., & Hsieh, Kuo-Cheng. Optimum cutting scores to discriminate groups of unequal size and variance. Journal of Applied Psychology, 1966, 50, 153-164. Strong, E. K. Vocational interests 18 years after college. Minneapolis: University of Minnesota Press, 19SS. Strong, E. K., Jr., & Campbell, D. P. Manual for Strong Vocational Interest Blank. Stanford, Calif.: Stanford University Press, 1966, Super, D. E., & Crites, J. O. Appraising vocational fitness. New York: Harper & Row, 1962. U.S. Bureau of the Census. U.S. census of the population: 1960. Detailed characteristics. United States Summary. (Final Report PC(l)-lD) Washington, D.C.: U.S. Government Printing Office, 1963. (Received June 7, 1973)