Control of Halo Error: A Multiple Regression Approach

Copyright 1985 by the American Psychological Association, Inc. 0021-9010/85/100.75

Journal of Applied Psychology 1985, Vol. 70, No. 3, 577-580

Control of Halo Error: A Multiple Regression Approach Avishai Henik and Joseph TzeJgov Department of Behavioral Sciences Ben-Gurion University of the Negev, Beer-Sheva, Israel In a recent article, Landy, Vance, Barnes-Farrell, and Steele (1980) suggested a method for excluding halo variance in rating scales. Their approach, however, may result in excluding true variance as well. The present work suggests a conceptualization of the halo effect in terms of a suppressor variable. Accordingly, a multiple regression approach for the treatment of halo variance is suggested.

The commonly accepted view of the dynamics of halo error assumes that there is some overall impression of the ratee's effectiveness (Holzbach, 1978). Recently, Landy, Vance, Barnes-Farrell, and Steele (1980) have shown that the halo effect may be reduced by using residual scores. Assume that p is a rating scale of some specific performance and g is a rating scale of general effectiveness. For the sake of simplicity let us assume that both p and g are standardized. The residual score used by Landy et al. (1980) may be written as:

This new score (x) is suggested as an estimate of p that is corrected for halo error. They suggested the usage of these residual scores in the context of validation rather than in the context of administrative decisions. They believe that x may be a better measure than p of some criterion to be predicted. One may imagine a situation in which p is suggested as a predictor, and in such a case x may be a better predictor than p. In any case, it seems that Landy et al.'s approach is oriented mainly towards validation. The purpose of the present work is to analyze this approach in the context of validation and to suggest an alternative approach that is more general.

Halo as a Suppressor Variable The ideas discussed in the present note apply to situations in which rating scales serve as predictors as well as to situations in which rating scales serve as criteria. It will be shown that in both cases halo can be conceptualized as a suppressor variable. From a statistical point of view

We wish to thank M. Ashkenazi, N. Shir, F. J. Landy, and two anonymous reviewers for their helpful comments. Requests for reprints should be sent to Avishai Henik, Department of Behavioral Sciences, Ben-Gurion University of the Negev, Beer-Sheva, Israel 84120.

as well as from the point of view of construct validity, the two cases are equivalent. We will begin our discussion with the situation in which the rating scales serve as predictors because the analysis in this case is relatively straightforward. The discussion of rating scales as criteria will be postponed to the second part of our note. Suppose we have a set of rating scales. Each rating scale PJ is designed to predict a specific behavioral criterion 9. Furthermore, let us assume that g is a general evaluation rating scale designed to capture halo error. For the sake of simplicity we will describe all relations of interest in terms of three variables, p, g, and c, and refer to the index j only when necessary. Assume that all the discussed variables are standardized and all the correlations among them are positive. Holzbach (1978) and Landy et al. (1980) described the general evaluation item g as having a substantial degree of covariance with the various specific items (i.e., rn + 0 for every p). They suggested that the variance contributing to this covariation is mainly the halo variance, which also elevates the intercorrelations among the specific rating scales. Landy et al. (1980) regressed the general evaluation rating (g) on the specific rating scales (PJ) and calculated residual performance ratings. They have shown that the intercorrelations among the residuals were smaller than the original intercorrelations among the ps. (see also Landy, Vance, & Barnes-Farrell, 1982). The exclusion of the unwanted variance from p via the computation of the residual scores makes sense if one assumes no systematic relation between the general rating scale g and the specific criteria! behavior c (i.e., ra = 0). It follows that the correction suggested by Landy et al. (1980) is based on the assumption that for every pair of a specific predictor and the corresponding criterion, the following relations hold: rK °£ 0, r№ = 0. This is of course an idealized situation but still this is the situation for which the suggested correction is best suited. When p and g are the two predictors in a multiple regression equation used to predict c, if the two conditions

577

578

SHORT

stated above hold, g constitutes a classical suppressor (Conger & Jackson, 1972; Horst, 1941; Tzelgov & Stem, 1978). Conger and Jackson (1972) suggested the following definition for such a suppressor: "A suppressor variable is here defined, in a manner consistent with the classical definition, as one wholly uncorrelated with a criterion but which, by virtue of a correlation with a predictor, improves the prediction of the criterion" (p. 581). That is, the suppressor variable is similar to the variable measuring halo, rcg = 0 and rK i= 0. The conceptualization of the item measuring halo as a suppressor gains more weight when one looks at the implied changes in validity due to the usage of this variable. Landy et al. (1980) suggested using the residual performance rating. That is, they suggested replacing x defined as: x = p — rp!g for p. In the context of validation, this manipulation would be useful if it increases the validity of prediction. That is, rcx should be greater than rcp (the original validity coefficient), r^ is nothing but the part correlation between c and p, excluding g from p; that is, rcx = r^f.g,. In other words, /•„ is the validity of that part in item p that is orthogonal to the general evaluation item, g. The desired relation between the original validity (i.e., r№) and the new validity, that is, rc(p.e}, can be stated as; r c(p-g> > rcp (assuming that all the correlations are positive). This relation was advocated as a definition for suppression situations (Holling, 1983; Velicer, 1978). That is, when the above relation (i.e., the part correlation greater than the zero-order correlation) holds, g constitutes a suppressor variable. This definition of suppression does not require that rcg = 0. Hence, g may constitute a suppressor variable even when r^ + 0 (Conger, 1974; Tzelgov & Stern, 1978). It follows that, contrary to the implicit assumption of Landy et al. (1980), g need not be totally independent of c in order to capture halo variance. From our discussion, it follows that halo variance is that part of the variance common to p and g that is criterion irrelevant. The function of a suppressor variable is to improve prediction by partialing out such variance. Thus, if g captures halo variance, g should constitute a suppressor. It should be noted, however, that Velicer's definition of suppression is too restricted (Tzelgov & Henik, 1981). A more general definition of suppression was suggested by Conger (1974): "A suppressor variable is defined to be a variable which increases the predictive validity of another variable by its inclusion in a regression equation" (p. 36). According to this definition, g acts as a suppressor whenever the following relation holds: ft. >

fcf

where ftp is the optimal least-squares weight of p

NOTES

in a multiple regression equation designed to predict c by p and g. (See Tzelgov & Henik, 1981, for further discussion of this point.) It follows that if a general evaluation item g captures halo variance, it should act as a suppressor variable. As we have shown, however, the best definition of suppression situation is within the context of multiple regression. Thus the multiple regression approach should be preferred as a way to treat halo variance. A Multiple Regression Approach We suggest that instead of partialing out g by the use of residual scores, one should try to predict the specific criterion c by a linear combination of the specific predictor p and the general evaluation item K- In this context one usually uses a multiple regression equation

The /3s are the optimal least-squares weights of the two predictors (i.e., p and g). The measure for the quality of prediction is the multiple R, or Rc.pg. When g is a classical suppressor, that is, when it has no correlation with the criterion (i.e., ra = 0), and an above-zero correlation with the other predictor (i.e., rpg > 0) the multiple R is equivalent to the part correlation. That is, Rc.ps — rc(f-g} (Conger & Jackson, 1972). In this case g contributes to prediction only through the subtraction of the irrelevant (halo) variance from the specific performance variable (p). Hence, the correction suggested by Landy et al. (1980) is equivalent with the multiple regression solution. When these ideal conditions of the classical suppressor do not exist (i.e., when rcg > 0), g may still constitute a suppressor variable (Conger, 1974; Tzelgov & Stern, 1978). In this case part of gs variance is related to the criterion (c) and part is not related to the criterion but only to the other predictor (p). The relative amounts of those two components determine if g constitutes a suppressor variable. Thus, if the variance shared by p and g (that is orthogonal to c) is higher than the variance shared by g and c, g serves as a suppressor variable (for a more elaborate discussion of this point see Darlington, 1968). In any suppression situation, the /3s of the suppressed variables (e.g., p) are larger than the corresponding zero-order validities. Thus, in these situations, the relative weight of the suppressed predictor is raised due to the inclusion of a suppressor variable (e.g., g). This fact is in accordance with the idea of the elimination of irrelevant (halo) variance from the predictors. Of course, it is easy to envision situations in which g would not constitute a suppressor at all but still has a systematic relation with the criterion. In such a situation

SHORT NOTES the inclusion of g in the regression equation would improve prediction. In the last two situations (when g is not a classical suppressor) the correction suggested by Landy et al. (1980) results in throwing away true variance and in a reduction of the efficiency of prediction. This is reflected in the fact that whenever rcg £ 0 the following relation holds:

The validity that results from Landy et al.'s correction is rc(f.e), whereas the validity that results from the application of a multiple regression is *,.„. To summarize, the relation between multiple correlation and part correlation is given by

The multiple correlation is equivalent to the part correlation when classical suppression exists. Only in this situation is the suggestion of Landy et al. (1980) as good as the multiple regression approach. Thus, in order to use the correction procedure suggested by Landy et al. (1980), one has to make sure that the general evaluation item is equivalent with the classical suppressor variable. Otherwise, an application of this correction will result in throwing true variance (Murphy, 1982). In fact, Landy et al. (1980) are aware of this problem: "It is difficult to determine what is being partialed out, true score or error score" (p. 505). In contrast, the use of the multiple regression approach would take care of both the elimination of halo variance and the inclusion of true, criterion related, variance in prediction. Thus, instead of employing the corrected p as a predictor, one should use the optimal linear combination of p and g. Rating Scales Used as Criteria In the process of test validation rating scales are frequently used as criteria rather than predictors. In such cases p is usually a psychological test designed to serve as predictor, whereas c is a specific criterion to be predicted by p. In such situations c may be the result of peer or superior rating, g is a rating scale of general effectiveness designed to capture halo error. Multiple regression can be employed in order to estimate the correlation between p and the linear combination of g and c. The purpose of this computation is to decide if g captures halo variance. In such a case ft—the least square weight of c—will be higher than rcp, which implies that g contributes by partialing out the unwanted (halo) error from c. Furthermore, in such a case Rf.cs rather than rcp is the best validity estimate. On the other hand, if ft. is lower than rcp, g does not capture halo and

579

should not be included in the criterion set. Thus, g should be included in the criterion set only if it constitutes a suppressor variable. The above application of multiple regression may seem strange to psychologists used to applying multiple regression for linear combinations of predictors rather than criteria. Note, however, that multiple regression is employed here as a special case of canonical correlation. Therefore, the approach suggested above is applicable not only when p stands for a single predictor but also when it stands for a set of predictors, since canonical correlation is designed to estimate the relations between linear combinations. Recently, Holling (1983) suggested a way to identify suppression situations in linear combinations. His work was based upon Velicer's (1978) restricted definition of suppression. Tzelgov and Henik (in press) extended his approach to fit the more general definition of suppression put forward by Conger (1974). Thus, it is possible to reveal whether g is a suppressor variable when included in the criterion set. In particular, g will be defined as a suppressor variable if its inclusion in the criterion set results in the elevation of the regression weight of c. If g is not a suppressor, one may be interested in excluding it from the criterion set in order to get a "purer" description of the criterial performance.

Conclusions We showed that when rating scales are used in the context of validation the multiple regression approach should be preferred over the simple technique suggested by Landy et al. (1980). The partial correlation approach as applied in the context of validation is based on the implicit assumption that when a general performance rating scale captures halo error, it acts as a classical suppressor. We have started with the conceptualization of halo error as a suppression effect, which in turn leads to the application of multiple regression. This approach maximizes prediction on the one hand and enables one to reveal halo variance on the other. It is possible to apply this approach both when the rating scales are used as predictors and when they are used as criteria. Furthermore, when one uses this approach one must assume nothing in advance about the relation between the general evaluation variable and the other variables. References Conger, A. J. (1974). A revised definition for suppressor variables: A guide to their indentification and interpretation. Educational and Psychological Measurement, 14, 35-46.

580

SHORT NOTES

Conger, A. J., & Jackson, D. N. (1972). Suppressor variables prediction, and the interpretation of psychological relationship. Educational and Psychological Measurement, 32, 579-599. Darlington, R. B. (1968). Multiple regression in psychological research and practice. Psychological Bulletin, 69, 161-182. Holling, H. (1983). Suppressor structure in the general linear model. Educational and Psychological Measurement, 43, 1-9. Holzbach, R. (1978). Rater bias in performance ratings: Superior, self, and peer ratings. Journal of Applied Psychology, 63, 579-588. Horst, P. (1941). The role of prediction variables which are independent of the criterion. In Horst, P. (Ed.): The prediction of personal adjustment. Social Science Research Bulletin, 48. 431-436. Landy, F. J., Vance, R. J., Barnes-Farrell, J. L., & Stesle, J. W. (1980). Statistical control of halo error in performance ratings. Journal of Applied Psychology, 65, 501-506. Landy, F. J., Vance, R. J., & Bames-Farrell, J. L. (1982). Statistical control of halo: A response. Journal of Applied Psychology. 67, 177-180.

Murphy, K. R. (1982). Difficulties in the statistical control of halo. Journal of Applied Psychology, 67, 161-164. Tzelgov, J., & Henik, A. (1981). On the differences between Conger's and Veneer's definitions of suppressor. Educational and Psychological Measurement, 41, 10271031. Tzelgov, J., & Henik, A. (in press). A definition of suppression situations for the general linear model: A regression weights approach. Educational and Psychological Measurement. Tzelgov, J., & Stern, I. (1978). Relationship between variables in three variable linear regressions and the concept of suppressor. Educational and Psychological Measurement, 38, 325-335. Velicer, W. F. (1978). Suppressor variables and the semipartial correlation coefficient Educational and Psychological Measurement, 38, 953-958.

Received August 27, 1984 Revision received December 18, 1984