Using Hierarchical Linear Models to Examine Moderator Effects ...

ORGANIZATIONAL Davison et al. / PERSON-BY-ORGANIZATION RESEARCH METHODS INTERACTIONS

Using Hierarchical Linear Models to Examine Moderator Effects: Person-by-Organization Interactions MARK L. DAVISON NOHOON KWAK YOUNG SEOK SEO JIYOUNG CHOI University of Minnesota

A cross-level interaction is said to occur when the effects of client or employee characteristics interact with organizational characteristics to influence an employee or client outcome variable. Hierarchical linear modeling (HLM) is briefly described, particularly as it applies to the study of cross-level interactions. HLM is then compared to moderated multiple regression (MMR). An HLM model incorporating cross-level interactions is illustrated with data from a study of test validity across organizational units. An HLM model for person-organization congruence is then described. As compared to MMR, HLM can more readily handle large numbers of organizations. By increasing the number of organizations that can be studied, HLM should increase the power of the designs that researchers can use.

When the behavior of individuals within organizations is studied, the data often have a nested structure. Individuals, usually employees or clients, are nested within organizations. Individuals are said to constitute the sampling units at the first and lowest level of the nested hierarchy. Organizations are said to constitute the sampling units at the second level. In such circumstances, the behavior of the individual can be a function of person characteristics (e.g., gender, job satisfaction), organizational characteristics (e.g., economic sector, cohesiveness), or the interaction of organizational and individual characteristics. Such interactions are called cross-level interactions because they involve units at two levels of the nested hierarchy. When the person-level predictor variable is continuous, the organizational variable in the cross-level interaction is called a moderator variable of the sort commonly studied with moderated multiple regression (MMR).

Authors’Note: Correspondence concerning this manuscript should be sent to Mark L. Davison, Department of Educational Psychology, University of Minnesota, 178 Pillsbury Dr. S.E., Minneapolis, MN 55455; e-mail: [email protected]. This research was funded by the Minnesota Department of Children, Families, and Learning through its support of the Office of Educational Accountability, University of Minnesota. Organizational Research Methods, Vol. 5 No. 3, July 2002 231-254 © 2002 Sage Publications

231

232

ORGANIZATIONAL RESEARCH METHODS

The focus of this paper is the use of hierarchical linear models (HLMs) for the purpose of studying such cross-level interactions. The HLM and the MMR approaches will be compared. To employ HLM, one must begin by formulating a model appropriate to the research question. Using achievement test data, we illustrate the formulation of HLM appropriate to questions of test validity and utility. We also describe an HLM formulation appropriate to questions regarding person-organization congruence. In education, “hierarchical linear models” is the term used to encompass analyses of designs with people nested within organizations (classrooms, schools, or districts), and HLM (Raudenbush, Bryk, Cheong, & Congdon, 2000) is the term we use throughout this paper. In health science research, however, they are often called mixed effects models. In that vein, the SAS (1996) analysis described by Singer (1998) is called PROC MIXED. In economics, they are often called random coefficient regression models. Multilevel models is another name for the same general class of analyses. Organizational Studies, Organizational Variables, and the Organizational Factor

For our purposes, an organizational study is any study that involves multiple organizations broadly defined. An organization is an enduring work unit composed of multiple individuals. By this definition, a department or work-group might constitute the “organizations” in a study. An organizational study becomes a multilevel study when multiple observations are taken within each organization. Because we are interested in person-by-organization interactions, the studies of interest are those in which the several observations within the organization correspond to different individuals. In the discussion below, the term “organizational factor” is a nominal variable designating the organizations in the study. It is an organizational factor in the ANOVA sense with one level for each organization. In HLM, the organizational factor is assumed random; i.e., organizations are assumed randomly sampled from a large population of organizations. As we use the term here, an “organizational variable” is any variable that is a constant for every individual within a given organization but that varies across organizations (e.g., location, size, cohesiveness). Employee ratings of (perceived) organizational climate might be an organizational variable to an organizational theorist in that the climate being rated is central to organizational theory. Because the ratings can vary across employees within an organization, organizational climate ratings would be an individual (level 1) variable, not an organizational variable, within the HLM framework. The average rating of climate by the employees of an organization would fit our definition of a level 2 organizational variable. The mean rating is a constant within an organization but can vary across organizations. Some organizational variables are aggregates of person scores (e.g., mean climate rating) and others are not (e.g., location). Our purely statistical definition serves to describe the central features of a level 2 variable in HLM, but we do not claim it is sufficient for purposes of organizational theory. For example, James, Joyce, and Slocum (1988) insist that there must be some consensus among employees about organizational climate before the mean rating can be considered a characteristic of the organization.

Davison et al. / PERSON-BY-ORGANIZATION INTERACTIONS

233

Just as one can distinguish between people per se and individual difference variables at level 1, one can distinguish organizations per se (levels of the organization factor) and organizational variables at level 2. Persons and organizations are the units sampled at level 1 and level 2 respectively. Individual difference variables and organizational variables are the measurements taken on those units. A cross-level interaction is the interaction of an individual difference variable at level 1 with an organizational variable at level 2. Selected Organizational Studies Using HLM

A number of cross-level interaction studies using HLM have appeared in the management literatures. Two recent studies have focused on work group cohesiveness (Griffin, 1997; Kidwell, Mossholder, & Bennett, 1997). Kidwell et al. found that as the work group’s cohesiveness increased, the effect of employee job satisfaction on courtesy behavior increased. In other words, work group cohesiveness moderated the relationship between employee job satisfaction and courtesy. Congruence between individual and group variables (e.g., individual and group goals) has been the subject of papers by Pervin (1989) and Vancouver, Millsap, and Peters (1994). In a later section of this paper, we present a squared Euclidean distance HLM model of congruence, one that intrinsically includes a cross-level interaction. Kreft and de Leeuw (1994) and Haberfeld, Semyonov, and Addi (1998) found evidence for cross-level interactions in their studies of gender differences in employee compensation. Kreft and de Leeuw found that the size of the gender gap in pay varied by type of organization (e.g., retail vs. educational vs. commercial). Haberfeld et al. (1998) studied occupational factors associated with the size of the male/female earnings differential in Israel. They found an interaction between gender (individual characteristic) and the status of an occupation, the size of the occupation, and the complexity in working with people. Specifically, gender-based differentials were smaller in high status occupations, occupations with a large number of workers, and occupations involving a high degree of complexity in working with people. As a tool for investigating cross-level interactions, our discussion of hierarchical linear models will be limited to just two levels, people nested within organizations. HLM analyses have been extended to more than two levels in the statistical literature on HLM and in the various computer programs. We begin with a brief description of HLM. Readers are referred to other sources in the organizational literature and the statistical literature for more details on the method (e.g., Bryk & Raudenbush, 1992; Griffin, 1997; Hofmann, 1997; Hofmann & Gavin, 1998). Following this brief general discussion of HLM, we turn to a comparison of the HLM and moderated multiple regression approaches to the study of cross-level interactions. Next, we turn to an extended analysis that uses the same data to address three different research hypotheses concerning the criterion-related validity of a test. In the extended analysis, we ask whether the validity of the test remains constant across organizations, whether the predicted Y value for each value of X remains constant across organizations, and whether the utility of the test remains constant across organizations. Finally, we describe an HLM model for the study of person/organization congruence, a model that implicitly involves a cross-level interaction.

234


Models and Analyses In HLM with two levels, each level is represented by its own equations. In a two-level analysis with people nested within organizations and a single predictor at level 1, the level 1 (person-level) model is: Yij = β0j + β1jXij + rij, (level 1)

(1)

where β0j is the intercept for the jth organization adjusted for the effect of person-level characteristic Xij; β1j is the slope in organization j, and rij is the residual at level 1 with a mean of zero and a variance of σ2. In ordinary regression, both the slope and intercept parameter are typically assumed fixed across organizations (i.e., homogeneity of regression). In the model above, the intercept and slope are allowed to vary across organizations. Hence, the organization subscript j is placed on the slope and intercept terms. Because the slope and intercept coefficients are allowed to vary, such models are sometimes called random coefficient models. Rather than estimating a single slope and a single intercept parameter, HLM estimates the mean and the variance of the intercept distribution and the slope distribution. τ00 and τ11 designate the estimated variances of the level 1 intercepts and slopes, respectively, across organizations. A chi-square statistic can be used to test the significance of the intercept variance estimate, τ00 , and the slope variance estimate, τ11. In words, the slope variance null hypothesis, H0: σ2(β1j) = 0, states that there is no variation in the slope parameter across organizations. If the null hypothesis H0: σ2(β1j) = 0 is retained, then there seems little point in searching for cross-level interactions (i.e., organizational moderator variables). Cross-level interaction terms in the level 2 equations (Equation 2b below) serve to account for variation in the slope parameter β1j. Parsimony dictates that we not add cross-level interaction terms to account for variation in β1j unless the data first indicate that β1j varies across organizations. Hence, the data provide evidence for a possible, but unspecified, cross-level interaction only if H0: σ2(β1j) = 0 can be rejected. If neither the slope nor the intercept vary across organizations, then there is no reason to add terms involving organizational variables to account for variation in slope or intercept. A full multilevel analysis is unnecessary (see de Leeuw & Kreft, 1995). Therefore, the analysis will often begin with an inspection of the estimates τ00 and τ11 as well as tests of the hypotheses H0: σ2(β0j) = 0 and H0: σ2(β1j) = 0 to determine whether a full multilevel analysis is required. If, however, the slope and intercept do vary across organizations, then one can move to the next step, accounting for variation in slope and intercept parameters across organizations by adding organizational variables to level 2 equations as below: β0j = γ00 + γ01Wj + u0j (level 2)

(2a)

β1j = γ10 + γ11Wj + u1j (level 2),

(2b)

where Wj is an organizational variable hypothesized to account for variation in slopes and intercepts across organizations. A cross-level interaction is said to exist between person-variable X and organizational variable W when the effect γ11Wj in Equation 2b


235

is nonzero. Put another way, a cross-level interaction exists between X and W when W accounts for variation in β1j across organizations. In Equations 2a and 2b, γ00 and γ10 represent the average intercept and slope respectively across organizations (given Wj), γ01 and γ11 are level 2 regression weights, and the level 2 residuals u0j and u1j have the following mean vector and variance-covariance matrix:  u0 j   0  τ 00   ~ N   ,  u  1 j  0  τ10

τ 01   . τ11  

(3)

The (2 × 2) matrix on the far right of Equation 3 contains the “covariance components” of the model, the variances and covariances of the residual terms in the model. As compared to a single-level analysis, HLM differs in three major respects. First, the slope and intercept terms are allowed to vary across organizations. Homogeneity of regression lines is a hypothesis to be tested, not simply an assumption. Parameter estimates include the means of the intercept distribution, γ00, and the slope distribution, γ10, along with estimated standard errors and degrees of freedom for these estimates. As in ordinary regression, a t-statistic can be used to test the significance of the level 2 intercepts, γ00 and γ10, and slopes, γ01 and γ11. Second, HLM includes additional parameters corresponding to the variances of the intercepts and slopes across organizations, τ00 and τ11. The hypotheses H0: σ2(β0j) = 0 and H0: σ2(β1j) = 0 can be tested to evaluate whether the intercept and slope vary across organizations. Third, when the intercepts and slopes vary across organizations, organizational variables are entered into level 2 equations to account for the intercept variation and the slope variation across organizations. HLM includes a measure of model fit, the deviance statistic that can be used to compare the fit of two models if one is hierarchically embedded in the other. The deviance statistic is a measure of fit for the covariance components of the model. The difference in the deviance statistics for two nested models will be asymptotically distributed as chi-square with degrees of freedom for the difference in deviance statistics equaling the difference in their respective numbers of estimated parameters in the covariance component of the two models. The number of parameter estimates equals the number of variances and covariances in Equation 3 plus an additional parameter corresponding to the variance of the residuals in Equation 1. That is, if there are R random coefficients in the level 1 model, then the number of estimated parameters in the covariance component of the model will be (1/2)R(R + 1) + 1. The deviance statistic equals –2.0 times the log of the likelihood function, the same fit statistic often used in structural equations modeling under the name “goodness-of-fit” rather than “deviance.” In HLM, particularly with complex models, the deviance statistic is used in place of the squared multiple correlation as a measure of fit because the various sources of error do not add together in a neat fashion that would give the squared multiple correlation a ready interpretation (Kreft, 2000). Given the default HLM option for restricted maximum likelihood estimation (rather than full maximum likelihood), the deviance statistic is primarily useful in deciding whether (a) specifying a coefficient at level 1 as random (rather than fixed) improves fit and (b) whether adding a predictor as a random effect to the level 1 equation improves fit. The deviance statistic seems less useful, however, in deciding whether to add predictors to the level 2 equations. The deviance statistic applies to the

236


covariance components of the model, and adding level 2 variables does not produce hierarchically nested models for the covariance components. To evaluate level 2 predictors, it seems wise to rely on statistical tests of the level 2 effect and examination of reductions in residual parameter variances (or standard deviations) that result from adding the level 2 predictor. In the specification of an HLM model, the researcher must be concerned about the centering of variables, particularly the level 1 variables. Centering refers to whether the variables are expressed as raw scores, as deviations about a grand mean, or as deviations about organization means. Various sources compare the centering options in more detail than space permits here (Aiken & West, 1991; Bryk & Raudenbush, 1992; Hofmann & Gavin, 1998; Kreft, de Leeuw, & Aiken, 1995). In our opinion, the most important consideration in choosing a centering option is the interpretation of the intercept term β0j and the slope term β1j. When no centering is used, β0j is the predicted value of Yij in organization j when the predictor(s) Xij = 0. When grand-mean centering is used, β0j is the predicted value Yij in organization j when the predictor(s) equals the grand-mean. When group-mean centering is used, β0j is the predicted value Yij in organization j when the predictor(s) equals the mean in group j. When the organization means vary, group-mean centering differs substantially from the other two options. To understand group-mean centering, it may help to consider the null hypothesis of equal intercepts in two organizations, one with a raw score mean of 10 on the predictor variable, and one with a raw score mean of 20. If group-mean centering is used, then the null hypothesis of equal intercepts is as follows. Is the predicted criterion score in group 1 when X = 10 (the predictor raw score mean in group 1) equal to the predicted criterion score in group 2 when X = 20 (the predictor raw score mean in group 2)? When organization means differ, the null hypothesis of equal intercepts may not be meaningful given group-mean centering and a predictor measured in the same units across organizations. In such cases, the equal intercept hypothesis involves an implicit comparison of intercept parameters, each of which is defined relative to a different group mean, the mean of group j. Group-mean centering changes the interpretation of the slope parameter as well as the intercept parameter. When group-mean centering is used, β1j becomes the slope of the regression line within group j. Often this within-group slope is the parameter of interest (Ostroff & Harrison, 1999). When grand-mean centering is used, β1j can be interpreted as the within-group regression slope only in certain special cases1 (Bryk & Raudenbush, 1992, p. 119), and therefore β1j is not always readily interpretable with grand-mean centering. When studying cross-level interactions, the centering decision is subsumed by a larger issue: whether to standardize the predictor and criterion variables within organizations. Whereas centering involves only the means, standardization involves both the means and variances. While the HLM literature has been vitally concerned with standardization of the mean predictor score within groups by group-mean centering, less attention has been paid to the issue of standardizing the variance within organizations. Standardizing variances within organizations changes the interpretation of the slope parameter. Irrespective of standardization, the slope is the change in Y associated with a unit change in X. If the variance of X is standardized separately by organization, then


237

“a one-unit change” in X has a somewhat different meaning in each organization, because a “unit change in X” is defined relative to each organization’s own standard deviation. The null hypothesis σ2(β1j) = 0 involves an implicit comparison of slope coefficients, each of which is expressed relative to the standard deviation unit in its respective organization. Not only must the researcher be concerned about variable centering and standardization, the researcher must also be concerned about model misspecification with an attendant catch-22. Inclusion of too many variables in the model can lead to severe multicollinearity. Inclusion of too few may lead to misestimation of some parameters. HLM can be used to address some of the same research questions as MMR. However, as described below, implementation of MMR with 2-level data is not so simple as it might at first seem.

Moderated Multiple Regression HLM can be used to divide the study of cross-level interactions into two questions. First, does the slope coefficient vary across organizations and thereby suggest the possibility of some unspecified cross-level interaction? Second, if slopes vary, what organizational variable(s) might explain or account for that variation? MMR offers an alternative approach to the study of these same questions. In this section, we compare two MMR models to their HLM counterparts. The first MMR model is analogous to an HLM model with one level 1 predictor and no level 2 predictors. It addresses the question “Do slopes vary across organizations?” The second is analogous to an HLM model with one level 1 predictor and one level 2 predictor. If the data suggest that slopes vary, this second model addresses the question of whether a specific level 2 predictor (e.g., a specific cross-level interaction) can account for that variation. To make the models more concrete, we will develop them as they would be applied to a specific situation. Consider a situation in which people and organizations have been sampled randomly. There is one level 1 predictor variable, X. There is one level 2 predictor, a dichotomous indicator of economic sector: retail vs. manufacturing. For simplicity, assume there are only four randomly sampled organizations, two in the retail sector and two in the manufacturing sector. In this design, organizations are nested within sectors. The dependent variable Yij is the criterion score for person i in organization j. Any MMR analysis of these data must address three issues not addressed by a simple, unmoderated regression equation: Yij = β0 + β1Xij + eij .

(4)

This model assumes homogeneity of regression lines in that it assumes the same slope and the same intercept for every organization. It assumes that errors are independently and identically distributed, when in fact errors may be independent across but not within organizations. This nonindependence can lead to underestimation of regression coefficient standard errors (Kreft, 2000), the standard errors used to test whether those coefficients differ significantly from zero.

238


Table 1 Sources of Variance in Moderated Multiple Regression Models (Equations 6 and 8) With and Without a Level 2 Predictor Equation 6: No Level 2 Predictors Source

Equation 8: One Level 2 Predictor Source

X — O — X×O Residual

X S O(S) X×S X × O(S) Residual

MMR Model with One Level 1 and No Level 2 Predictors

To create this model, we need to introduce dummy variables that code the four organizations. Let Djk (k = 2, 3, 4) be defined as follows: Djk =1 for the kth organization, = 0 otherwise.

(5)

In the language of HLM, Djk is not a level 2 variable. The variable Djk codes organizations per se (levels of the organization factor) and therefore is not a true level 2 organizational variable. With these new variables, we can create the following MMR model Yij = β0 + β1Xij + β2Dj2 + β3Dj3 + β4Dj4 + β5Dj2Xij + β6Dj3Xij + β7Dj4Xij + eij

(6)

While this model does not contain a unique intercept term, β0j for each organization, a separate intercept for each organization is implicitly included as β0j = β0 + β2Dj2 + β3Dj3 + β4Dj4. Similarly, while there is no explicit slope term βj for each organization, a separate slope for each organization is implicitly included as βj = β1 + β5Dj2 + β6Dj3 + β7Dj4. By implicitly including separate slope and intercept coefficients for each organization, this model relaxes the homogeneity of regression assumption. By allowing the slope and intercept to vary across organizations, it becomes a random coefficient model, because the slope and intercept coefficients now vary randomly across organizations. Through the additional terms, the independence assumption is altered; errors are now assumed independent between but not within organizations. The organization specific intercept β0j = β0 + β2Dj2 + β3Dj3 + β4Dj4 is used to account for the nonindependence of errors within organizations, just as in the HLM model. Viewed from an ANOVA design perspective, this is a design with two factors, X and organizations. The two factors are completely crossed.2 Table 1 shows the sources of variance in this design. Both the organization main effect, O, and the X-by-O interaction would be tested using the mean square residual as the error term (Maxwell and Delaney, 1990, p. 431). In Equation 6, the main effect for X is represented by the term β1Xij. The main effect for organizations is represented by the terms (β2Dj2 + β3Dj3 + β4Dj4). The hypothesis of no organization main effect is equivalent to the HLM hypothesis σ2(β0j) = 0. If the


239

organization main effect hypothesis could not be rejected, then there is no need to enter true level 2 organizational variables to account for intercept differences among organizations, because the data disconfirm the existence of such intercept differences. The Xby-O interaction in Equation 6 is represented by the moderator terms (β5Dj2Xij + β6Dj3Xij + β7Dj4Xij). The hypothesis of no interaction is equivalent to the HLM hypothesis σ2(β1j) = 0. If the interaction hypothesis is not rejected, then there is no need to enter organizational variables to account for slope differences among organizations, because the data disconfirm the existence of such slope differences.3 Because not every value of X will appear an equal number of times in each organization, the design is unbalanced. Therefore, the researcher will want to add terms and test hypotheses hierarchically, starting with the X main effect first represented by the term β1Xij, followed by the main effect of organizations represented by the terms (β2Dj2 + β3Dj3 + β4Dj4), and followed by the X-by-organization interaction represented by the terms (β5Dj2Xij + β6Dj3Xij + β7Dj4Xij). The model in Equation 6 is the MMR equivalent of an HLM model with one level 1 predictor and no level 2 predictors. It would be used to confirm or disconfirm the existence of an organizational main effect and an X-by-organization interaction. If either the organization main effect or the X-by-organization interaction null hypothesis is rejected, then true level 2 variables could be included to investigate organizational variables that could account for the variation across organizations in slopes or intercepts. In the next model, we assume that both an organization main effect and an X-by-organization interaction exist, and we include a true level 2 variable to account for those effects. MMR Model with Level 1 and Level 2 Predictors

To test for an effect of the true level 2 variable, economic sector, Equation 6 must be rewritten in terms of organizational variables that include an indicator of economic sector. Let Wj = 1 if organization j is in the retail sector, and let Wj = 0 if organization j is in the manufacturing sector. Furthermore, let Gj3 = 1 for the first organization in the manufacturing sector, and = 0 otherwise.

(7)

Similarly, let Gj4 = 1 for the first organization in the retail sector, and = 0 otherwise.

Then Equation 6 can be reparameterized as Yij = β0 + β1Xij + β2Wj + β3Gj3 + β4Gj4 + β5WjXij + β6Gj3Xij + β7Gj4Xij + eij

(8)

From the ANOVA perspective, Equation 8 is the model for a design with three factors: X, organizations, and sectors. Variable X is crossed with organizations that are nested within sector: X × O(S). The sources of variance for this design are given in the right portion of Table 1. In Equation 8, the term β1Xij corresponds to the main effect of X. The term β2Wj corresponds to the main effect of sector. The next two terms (β3Gj3 + β4Gj4) model the

240


main effect of organizations, O(S). The fifth term β5WjXij models the cross-level interaction between sector and X: X × S. And the next two terms (β6Gj3Xij + β7Gj4Xij) model the interaction of organizations with X: X × O(S). Equation 8 is a reparameterization of Equation 6. Hence, the two equations will account for the same amount of variance in Yij. The mean square residual for the two models will be the same. Reparameterizing the model partitions the sum of squares for organizations, SSO, into two components: one component that represents variation in the intercepts that can be accounted for by sector, SSS, and a second that represents variation in the intercepts that cannot be accounted for by sector, SSO(S): SSO = SSS + SSO(S). Because organizations are nested within sectors, the appropriate mean square (MS) error for testing the sector main effect is MSO(S) (Maxwell and Delaney, 1990, p. 448). Reparameterizing the model also partitions the sum of squares for the X-by-organization interaction, SSX × O, into two components: one component that represents variation in the slopes that can be accounted for by the X-by-sector interaction, SSX × S, and a second that represents variation in the slopes that cannot be accounted for by the sector interaction, SSX × O(S) : SSX × O = SSX × S + SSX × O(S). Because organizations are nested within sectors, the appropriate mean square error for testing the X-by-sector interaction is MSX × O(S) (Maxwell and Delaney, 1990, p. 448). In nested MMR designs, researchers must be careful to test the level 2 main effect and the cross-level interaction effect for the organizational variable with the appropriate error terms. The full multilevel model is composed of three equations (Equations 1, 2a, and 2b), each with its own separate error term. Each of these three error terms corresponds to a different sum of squares in Table 1: SSRes corresponds to the level 1 error term in Equation 1, SSO(S) corresponds to the level 2 error term in Equation 2a, and SSX × O(S) corresponds to the level 2 error term in Equation 2b. In each of these three equations, an effect in the equation must be tested with the error term for that equation. Unfortunately, researchers have sometimes used an MMR analysis based on a model simpler than Equation 8. The simpler analysis errs by ignoring the level 2 organizational factor, and as a result it fails to provide estimates of all needed error terms. The simpler model does not include dummy codes for organizations. As applied to our example, the simpler analysis omits the terms involving Gj3 and Gj4 from the model in Equation 8. As a result, the simpler model leads to incorrect tests of level 2 hypotheses because it fails to provide estimates of level 2 error terms. The main effect null hypothesis for the organizational variable W should be tested with the MSO(S) as the error term, but MSO(S) is not estimated in an analysis based on the simpler model. Likewise, the X-by-W interaction should be tested using the MSX × O(S) as the error term, but the MSX × O(S) is not estimated in the analysis based on the simpler model. The simpler model leads to an incorrect test of the level 2 effects based on the level 1 MSRes as the error term (Maxwell and Delaney, 1990, p. 448). Even with only four organizations, the MMR equations with cross-level interactions are quite complex. If there are np predictors and no organizations, there will be no(np + 1) terms in the MMR equations. The number of terms increases as no, np, or both increase. As the number of organizations increases, the number of terms in the MMR can begin to exceed hardware memory limits, software array size limits, or the researcher’s patience with dummy coding. Therefore, the MMR approach is limited to designs with small numbers of organizations.


241

Table 2 Descriptive Statistics for Urban Schools and Non-Urban Schools

Number of students Number of schools 3rd grade mean 3rd grade SD 5th grade mean 5th grade SD

Urban Schools

Non-Urban Schools

6,350 124 1,304.8 192.7 1,365.6 238.2

47,629 757 1,431.5 167.7 1,520.2 211.5

Note. Means and standard deviations (SD) are in unstandardized, scale score units.

In contrast, the HLM approach can accommodate even large numbers of organizations. Just as any ANOVA design requires an adequate sample size, an HLM analysis requires a moderately large sample of organizations (Bryk and Raudenbush, 1992, pp. 222-224). Bryk and Raudenbush (p. 211) suggest at least 10 organizations with just one level 2 equation containing one predictor, with the minimum number of organizations increasing as the number of level 2 equations increases and as the number of predictors in each level 2 equation increases. Kreft (2000) says the assumptions of HLM are best met when organizations are sampled from a large population of organizations. In testing either a main effect or a cross-level interaction involving a true level 2 variable using MMR, the power of the test will depend on the number of organizations, all other things being equal. But the MMR approach is limited to situations with a small number of organizations. Therefore, MMR seems impractical with designs that achieve greater power by virtue of including a large number of organizations. More research is needed on the power to detect moderator effects with HLM analyses, research that systematically varies the number of organizations and the number of people sampled within each organization as well as various artifacts (Aguinis and Stone-Romero, 1994).

Examples: HLM Selection Test Models In this section, we illustrate the application of HLM to three null hypotheses regarding a predictor test. In our example, the predictor test is a measure of third grade reading achievement. The criterion variable is a fifth grade reading test. Broadly speaking then, we are interested in how well one can predict fifth grade achievement using third grade reading scores. All three null hypotheses concern whether slopes remain constant across organizations. Each null hypothesis corresponds with one of three selection test models that we call the equal validity model, the equal prediction model, and the equal utility increment model. In this illustration, the level 2 organizations are schools. The level 1 unit is a student, a client of the school. Table 2 shows descriptive data for our example. Our data set is extremely large. It includes over 50,000 students in over 800 schools. In the number of individuals and organizational units, it is unrepresentative of most organizational research. However, the example illustrates that HLM can readily be applied to studies with large numbers of organizational units. In this example, we illustrate how several hypotheses about a selection test can be translated into questions about the slope and intercept parameters of a regression equation by standardizing the predictor and criterion variables in various ways.

242


Do Slopes Vary? Equal Validity, Equal Prediction, and Equal Utility Increment Models

The first model below is a validity generalization model. The study design, however, is not the usual validity generalization design. The usual validity generalization meta-analysis includes only level 2 validity data from several organizational units. A full HLM analysis requires level 1 data; that is, predictor and criterion data from applicants in multiple level 2 organizational units. A full HLM analysis would be most useful for a single validity study in which predictor and criterion data were gathered within several organizations. An example might be a military study in which the predictor test is correlated with criterion data from recruits assigned to several different military installations. The question, then, is whether the validity of the test generalizes across the several installations included in the single study. In a single study involving several organizational units, individual respondent level data would be available. If so, then HLM can be used to test three hypotheses about the selection test, each of which can be framed as a question about a cross-level interaction. Equal validity model: validity generalization. In our equal validity model, the null hypothesis states that the test’s validity coefficient remains constant across organizational units. According to this hypothesis, any variation in observed test validities across organizations is due to sampling error. In effect, this null hypothesis is what Schmidt and Hunter (1977, 1984) have called the “bare bones” model. As compared to the bare bones model, the full validity generalization model includes additional sources of variation in validity coefficients across studies (e.g., range restriction and measurement error in the criterion variable). If both the predictor test X and the criterion variable Y have been standardized to have mean 0.0 and variance 1.0 in each group j, the intercept term in Equation 1 becomes zero and the population slope parameter equals the mean uncorrected (for unreliability or restriction of range) population validity coefficient. That is, the slope parameter in organization j and the validity coefficient in organization j, ρxyj, are equivalent. Any test of a hypothesis about the slopes β1j is a test of the equivalent hypothesis about the validity coefficients. In fitting this bare bones validity generalization model, X and Y were both standardized to have mean 0.0 and variance 1.0 in each school. Table 3 summarizes the key results arising from fitting the equal validity model. Because the intercept is a fixed parameter, the unconditional model (with no level 1 predictor) could not be successfully fit to the data, so the deviance statistic for the unconditional model is not shown in Table 3. In fitting the equal validity model, the estimate of the intercept was zero to two decimal places as it should be because both the predictor and criterion have been standardized to have mean 0.0 (and variance 1.0) in each organization. Thus, the intercept must be zero in all organizations. The estimate of the mean slope (or mean validity) µ(β1j) was .77, which was significantly greater than 0.0 at the .01 level. This value is very close to the correlation between X and Y computed in the total sample, .79. The estimate of σ(β1j) was .01, suggesting that the slope parameter (or the validity) varies little across organizations. This estimate of the slope parameter was not significantly different from 0.0 at the .01 level. Hence, the equal validity (or equal slopes) hypothesis cannot be rejected. Because the slope variation is nonsignificant, there is no need to add level 2 variables to account for variation in the slopes/validities across organizations.


243

Table 3 Selection Test Models Examining Whether Slopes Vary Equal Validity Model Unconditional deviance Number of parameters Selection test model deviance Number of parameters Level 1 intercept µ(β0j) σ(β0j) Level 1 slope µ(β1j) σ(β1j)

— — 103,872 2 0.00 — 0.77** 0.01

Equal Prediction Model 730,289 2 680,277 4

Equal Utility Increment Model 709,719 2 682,348 4

1,500.88 32.37**

1,489.48** 88.24**

0.98** 0.07**

158.40** 22.66**

**p < .01.

If the null hypothesis of equal validities had been rejected in favor of the alternative, we could then have begun adding organizational variables to Equation 2b to account for the variation in the slopes (validities) across organizations. Schmidt and Hunter (1977) suggest three variables that could be added to Equation 2b: predictor unreliability, criterion unreliability, and restriction of range. That is, these psychometric factors might be built into Equation 2b to see if they account for validity variation across organizations. It might be argued that these psychometric factors are not organizational variables. Certainly they are not variables representing the structure, management, or organization of the unit. They may, however, be relatively enduring characteristics of an organizational unit. That is, because a unit is attractive to applicants, it may consistently draw a large number of applicants leading to a consistently low selection ratio. The criterion variable, such as personnel evaluations, may be consistently more reliable or display less restriction of range in one unit than another. While consistent differences among organizations on psychometric variables will almost certainly account for some variation in test validity across organizations, should it exist, other organizational variables may play a role. That is, organizational variables may account for variation in validity over and above that accounted for by psychometric factors, or organizational effects may be mediated by psychometric factors (e.g., organizational attractiveness to applicants may be mediated by psychometric selection ratio). Equal prediction model. In this model, the null hypothesis is that, for any predictor score, X, the expected score on the criterion variable Y′ is the same in every organization. For this hypothesis to be true, the regression line for X and Y must be the same for all organizations. That is, the intercepts β0j must be equal across all organizations, and the slopes β1j must be equal across all organizations. The hypothesis can be tested by fitting Equation 1 with X and Y expressed in raw score form and examining the estimates of σ(β0j) and σ(β1j). If both of the hypotheses, σ2(β0j) = 0 and σ2(β1j) = 0, are retained, then the hypothesis of equal regression lines, and hence equal predicted values Y′ for any given value of X is retained. Before fitting the model of Equation 1, we first fit the unconditional model (with no level 1 predictor) so we could compare the improvement in fit from adding X to the model. Table 3 shows the deviance statistic for the unconditional model. Table 3 also

244


shows results from fitting Equation 1 to the raw scores with grand-mean centering. To test the equal prediction model, grand-mean centering (rather than group-mean centering) must be used so that the null hypothesis σ2(β0j) = 0 corresponds to the equal intercepts hypothesis in the situation where all organizations have the same regression line.4 The difference in the deviance statistics for the unconditional and equal prediction models was 730,289 – 680,277 = 50,012, a statistic which should be asymptotically distributed as chi-square with 4 – 2 = 2 degrees of freedom. The improvement in fit from adding the predictor X is substantial and significant. The estimate of µ(β0j) was 1500.88. When, as in this example, X has been grand-mean centered, the intercept for organization j, β0j, is the predicted value Y′ in group j for a person with a predictor score X equal to the grand mean. The estimate of σ(β0j) was 32.37, which was significant at the .01 level. Thus, the hypothesis of equal intercepts across organizations is rejected. The estimate of µ(β1j) was .98, and is the estimate of the average slope across groups. The estimate of σ(β1j) was .07, which was significant at the .01 level. Thus, the hypothesis of equal slopes across organizations is also rejected. Because both the hypotheses of equal slopes and intercepts have been rejected, the hypothesis of equal regression lines in all organizations is untenable. The hypothesis of equal expected criterion performance across organizations for any given value of X is also untenable. Equal utility increment model. Another way to evaluate a selection test is to examine the increment in the average expected outcome when the test is used for selection as compared to the expected outcome based on random selection. This increment can be designated as ∆u. According to Cronbach and Gleser (1965; cited in Ghiselli, Campbell, & Zedeck, 1981), the increment has the following form when X and Y have a bivariate normal distribution and the predictor is standardized: ∆u = ρxyσyE(x′), where ρxy is the validity coefficient, σy is the standard deviation of the criterion value, and E(x′) is a quantity that depends on the selection ratio. Our equation gives the expected increase per person on the outcome variable before subtracting the cost per person, whereas Cronbach and Gleser (1965) and Ghiselli et al. (1981) use the symbol ∆u to designate the increase in utility net of cost. For a given selection ratio [i.e., a given value of E(x′)], the increment in utility depends on the quantity ρxyσy, a quantity that equals the slope of the regression line when X is standardized by group and Y is expressed in raw (or deviation) score form. Therefore, one can test the null hypothesis that, for any given selection ratio, the increment in utility remains constant across organizational units by testing the null hypothesis that ρxyσy remains constant across organizations. That is, one can test the null hypothesis that, for any given selection ratio, the increment in utility remains constant across organizations by testing the null hypothesis that the slope coefficient is a constant when X is expressed in standardized form and Y is expressed in raw (or deviation) score form. This model is intermediate between the equal prediction and equal validity models. It differs from the equal validity model only in that the criterion variable, Y, is expressed in raw score form. It differs from the equal prediction model only in that the predictor has been standardized to have mean 0.0 and variance 1.0 in each group. To fit this model, the predictor variable (but not the criterion) was standardized separately for each organization, rather than being standardized for the total sample. This model


245

may be preferred over the equal validity model when the criterion is expressed in meaningful units (dollars sold, days absent) and the researcher wants to see whether the test utility, expressed in that meaningful unit, is equal across all organizations holding the selection ratio constant. The deviance statistic for the unconditional model was 709,719 with two estimated parameters. Adding the predictor X to the model decreased the deviance statistic to 682,348 with four estimated parameters. This is a large and statistically significant reduction in the fit statistic. Both the estimates of the population mean intercept and slope were significantly greater than 0.0 at the .01 level. In testing the equal utility increment conjecture, the important hypothesis is H0: σ2(β1j) = 0. As shown in Table 3, the estimated standard deviation for the slope parameter was 22.66, which was significant at the .01 level. Consequently, we would reject the null hypothesis that, for a given selection ratio, the increment in utility would be the same across all organizations. As illustrated by these three models, the validity generalization hypothesis is a special case of a more general HLM model that includes an equal prediction model and an equal utility increment model as other special cases. The HLM model places the validity generalization hypothesis in a larger framework. Within that larger framework, the validity generalization hypothesis may not be the only hypothesis of interest concerning a selection test. To the question of whether slopes vary, one can get different answers depending on how the criterion and predictor(s) are standardized, as this example illustrates. These are not really conflicting answers, but rather different answers to different questions. The interpretation of the slope parameter varies depending on how the variables are standardized. Having illustrated how HLM can be used to test a hypothesis about whether slopes vary, we now turn to the second question: “If slopes vary, what organizational variables may account for that variation?” What Organizational Variable(s) Accounts for the Slope Variation?

By adding organizational predictors to the level 2 equation for the slope, our Equation 2b, HLM can be used to test hypotheses about specific organizational variables that may account for slope variation across organizations. For our example, we added a location variable, a dichotomous indicator of whether the school was located in an inner city or not: Wj = 1 if the school is located in an inner city area; Wj = 0 if not. We included this variable in both the level 2 equations for the intercept, Equation 2a, and for the slope, Equation 2b. Inner city location is thought by some to be associated with larger school size, a more bureaucratic structure, and a poorer climate. This is not to say that all inner city schools are large, bureaucratic, and suffering from a poor climate, but associations are hypothesized to exist. Therefore, location is of interest to education organizational theorists, because organizational variables may possibly mediate the relationship between school outcomes and location (Bryk, Lee, & Smith, 1990; Lee & Smith, 1997). Table 4 contains the results of this analysis. The first column shows results from a model with no level 2 predictors. It was the same as the equal prediction model of

246


Table 4 Accounting for Variation in the Slopes With Urban/ Non-Urban Location Variable: Equal Prediction Model With No Level 2 Predictors Deviance Number of estimated parameters Level 1 intercept µ(β0j) σ(β0j) Level 1 slope µ(β1j) σ(β1j) Level 2 intercept model Location effect Level 2 slope model Location effect

680,277 4 1,500.88** 32.37** 0.98** 0.068**

With Level 2 Predictors 680,179 4 1,501.13** a 30.59** 0.98** 0.065**a –28.86** 0.042**

a

The square of this quantity is the variance in the parameter unaccounted for by the location variable. **p < .01.

Table 3. The second column shows the results of adding the school location variable to both the level 2 intercept and level 2 slope models. Intercept Parameter. Given that we entered the location variable in uncentered form and the predictor variable in grand-mean centered form, the estimate of µ(β0j), 1,501.13, is the estimate of the average intercept for a non-urban school. That is, 1,501.13 is the expected criterion score for a student in the average, non-urban school whose predictor score falls at the grand mean. In the level 2 model without the location variable, the square of σ(β0j), 32.372 = 1,047.82, is an estimate of the intercept variance across organizations. In the level 2 model with the location variable, the square of σ(β0j), 30.592 = 935.75, is an estimate of the intercept variance unaccounted for by location. Eighty-nine percent of the intercept variance cannot be accounted for by the location variable (935.75/1,047.82 = 0.89), and 11% of the variance can be accounted for. Slope Parameter. The estimate of µ(β1j), 0.98, is an estimate of the average slope parameter in non-urban schools. Correspondingly, the estimate of σ(β1j), .065, quantifies the residual variation in the slope parameter across schools after controlling for location. Table 4 shows a significant effect of location on the slope parameter. Comparing the two estimates of σ(β1j) in Table 4, however, suggests that the effect is rather small. In the model with no level 2 predictor, the square of σ(β1j), .0682 = 0.004624, is an estimate of the slope parameter variance across organizations. In the model with a level 2 predictor, the square of σ(β1j), .0652 = 0.004225, is the slope variance unaccounted for by the location variable. The unaccounted for variance is 91% of the slope variance (.004225/.004624 = 0.91). Conversely, 9% of the slope variation across organizations was accounted for by the level 2 location variable. Regression Equations for Urban and Non-Urban Schools. In the level 2 intercept equation, the location effect estimate is –28.86, suggesting that the average intercept for urban schools is almost 30 points lower than that for non-urban schools. The aver-


247

1800

Fifth Grade Reading Score

1600 Urban 1400 Non-Urban 1200

1000

800 800

1000

1200

1400

1600

1800

Third Grade Reading Score

Figure 1: Predictor and Criterion Regression Lines for Students in Urban and Non-Urban Schools: Equal Prediction Model

age intercept for the urban schools can be computed by taking the average intercept for the non-urban schools (1,501.13) and adding the location effect (–28.86). This computation yields an estimate of the average urban intercept equal to 1,501.13 – 28.86 = 1,472.27. In the level 2 slope equation, the location effect is .042, suggesting that the average slope is steeper in urban than in non-urban schools. The average slope for urban schools can be computed by taking the average slope for non-urban schools (0.975) and adding the location effect (0.042). This computation yields an estimate of the average urban slope equal to 0.975 + 0.042 = 1.017. Using the average slope and intercept estimates for non-urban schools, the regression equation is Y ′ = .975X + 1,501.13 where X is expressed as a deviation about the grand mean. For urban schools, the regression equation becomes Y ′ = 1.017X + 1,472.27 where again X is expressed as a deviation about the grand mean. Figure 1 shows the regression lines for urban and non-urban schools based on the estimated average slope and intercept coefficients for each type of school. These lines reflect location differences in both intercepts (main effects) and slopes (X-by-W interaction). As scores on the predictor test increase, the difference in expected performance between urban and non-urban students decreases. In the three analyses above, we have gotten three answers to questions regarding cross-level interactions by standardizing the predictor and criterion in three different ways: standardizing both the predictor and criterion variables within organizations, standardizing neither, and standardizing only the predictor within organizations. In the first analysis, standardizing both variables within organizations to address an equal validity hypothesis, we could not reject the hypothesis of equal slope/validity coefficients across organizations, suggesting that no cross-level interaction need be posited to account for the data. The equal validity hypothesis was not rejected. In the second case, the equal prediction model, neither variable was standardized. We

248


rejected the hypotheses of equal slopes and intercepts across organizations and we found an organizational variable, urban vs. non-urban location, that accounted for some of the variation in slope parameters. In this second case, we rejected the hypothesis that the predicted criterion performance is constant across organizations given X, and we found an organizational variable to account for some of the differences. The data supported the existence of a cross-level interaction between initial student test scores and organizational location. In the third analysis of an equal utility increment model, we standardized only the predictor variable within organizations. We concluded that the slope coefficient varied across organizations, lending support to the conclusion that some unspecified cross-level interaction exists, but we did not identify any specific cross-level interaction that could account for the variation, and we rejected the equal utility increment hypothesis, but we did not identify a specific organizational variable to account for the variation of utility across organizations. As we have tried to illustrate with these analyses, the decision to standardize one or both variables within organizations is critical to the study of cross-level interactions. The decision changes the interpretation of the slope coefficient and thereby changes the nature of the research question.

A Weighted, Squared Euclidean Distance Model of Person/Organization Fit Hypotheses about selection tests are not the only ones that can be tested using models containing a person-by-organization interaction. A person-by-organization congruence hypothesis can be formulated in terms of an HLM model that includes personby-organization interaction terms. The HLM model incorporates a weighted squared Euclidean distance conception of congruence that is a generalization of the Euclidean distance D congruence measure employed by Vancouver and Schmitt (1991). Vancouver and Schmitt (1991) studied the effect of goal congruence on job satisfaction, organizational commitment, and intent to quit. They examined overall congruence, rather than congruence on specific goals. In what follows, we present an HLM for congruence on one or more specific issues. The model will be developed around congruence on a single issue. The generalization to more than one issue is presented in Appendix A. In this generalization, each issue can be incorporated into the model as a separate term, thereby minimizing interpretation problems that arise when several issues are confounded in a single congruence index (Edwards, 1995). As shown by Equations 11a-d below, our specification of the congruence model is quadratic and inherently includes a cross-level interaction. Thus, one way to test a person/organization congruence hypothesis is to implement HLM so as to test both the quadratic and cross-level interaction predictions. As before, let Yij be the outcome measure for person i in organization j; let Xij be the level 1 predictor variable; and let Wj be the level 2 organizational variable. For instance, in a study of the relationship between employee-supervisor attitude congruence and job satisfaction, the employees would be the people sampled at level 1. The work groups would be the organizations sampled at level 2. The level 1 variable Xij would be an attitude score of person i in work group j. The level 2 variable Wj would be an attitude score for the supervisor of work group j. A measure of job satisfaction for person i in group j, Yij, would serve as the criterion variable to be predicted from employee-supervisor congruence on the attitude measure.


249

In the equations below, we assume that Y has been scored so that high values of Y are associated with high levels of congruence, which means that high values of Y are associated with small squared Euclidean distances. According to the congruence model proposed here, Yij = β0j* + β1j(Xij – Wj)2 + rij β1j < 0.

(9)

This model is an adaptation from the scaling literature on external unfolding (Davison, 1976). In Equation 9, congruence is operationalized as a squared Euclidean distance (Xij – Wj)2 weighted by β1j. The greater the congruence (smaller the squared distance) between Xij and Wj, the greater the outcome variable Yij; for example, the greater the congruence between employee and supervisor attitudes, the higher the satisfaction score. Squaring the quantity in parentheses on the right side of Equation 9 and including residual terms for the level 2 equations yields the following: Yij = β0j + β1jXij2 + β2j Xij + rij β1j < 0,

(10a)

β0j = β0j* + β1j W j2 + u0j,

(10b)

β2j = –2β1jWj + u2j.

(10c)

and

Translating these equations into level 1 and level 2 forms analogous to Equations 1, 2a, and 2b yields Yij = β0j + β1jXij2 + β2j Xij + rij

(11a)

β0j = γ00 + γ01Wj2 + u0j,

(11b)

β1j = γ10 + u1j

(11c)

β2j = γ20 + γ21Wj + u2j

(11d)

and

where γ20 = 0 and γ21 = –2β1j. In the squared Euclidean distance conception of congruence in Equation 11a, the outcome variable Y is a linear function of X and X2. Equations 11b and 11d are the level 2 equations showing how the level 1 coefficients, β0j and β2j,vary as a function of the organizational characteristic W. Equation 11d contains the cross-level interaction. Equations 11a-d suggest testing the congruence hypothesis by fitting a series of models. The first would be the unconditional model, so that its deviance statistic could

250


be used as a baseline fit measure. The second model would include just the linear term involving X, but no organizational predictors at the second level. The congruence hypothesis would predict that this linear model fits better than the unconditional model and that the variance of the regression slopes is greater than 0. If these predictions are supported, the third model would add a quadratic term X2 (with no organizational predictors at the second level yet). Finally, the last model would add predictors W and W2 in level 2, as specified in Equations 11b and 11d. If this congruence model holds, the coefficient on Xij2 in Equation 11a and the coefficient on Wj2 in Equation 11b should both be negative (because β1j < 0) and equal. The coefficient on Wj in Equation 11d should be positive (because β1j is negative and therefore –2β1j is positive). As shown in Appendix A, the model in Equations 11a-d can be readily extended to include more than one dimension of congruence. The congruence model predicts that β1j < 0. If, however, β1j > 0, then the model becomes a complementarity model (Davison & Jones, 1976) in which the outcome variable increases as congruence decreases.

Conclusions HLM can be used to break the study of cross-level interactions into two questions: Do slope parameters vary across organizations suggesting the existence of some unspecified cross-level interaction? If so, what organizational variables may account for that variation in slope parameters? Designs that include both person- and organization-level variables must cope with three issues: the nesting of organizations within organizational variables, the possible heterogeneity of regression across organizations, and the possible nonindependence of errors within organizations. MMR can cope with these issues, but an appropriate MMR analysis can quickly become impractical as the number of organizations, the number of person-level predictor variables, or both increases. The organization sample size requirements and the assumptions of HLM require a moderate to large number of organizations randomly sampled from a population of organizations. Statistical power for detecting a cross-level interaction increases as the number of organizations increases, and MMR is impractical with a large number of organizations. Aguinis (Aguinis, 1995; Aguinis, Pierce, & Stone-Romero, 1994; Aguinis and Stone-Romero, 1997) has shown that MMR often has little power for detecting moderator effects. HLM permits researchers to incorporate large numbers of organizations into their designs, thereby enabling researchers to use more powerful designs in the study of person-by-organization interactions. Further research is needed comparing the power of MMR and HLM for detecting cross-level interactions in designs with varying numbers of people and organizations. With appropriate standardization of the predictor and criterion variables, there are three hypotheses concerning a selection test that can be addressed through an HLM analysis of slope coefficients: equal validity, equal prediction, and equal utility increment. Our HLM approach to validity is applicable to single validity studies encompassing several organizational sites and including level 1 data. It is not applicable to meta-analyses (e.g., Erez, Bloom, & Wells, 1996; Ostroff & Harrison, 1999) that encompass several studies but no level 1 data. Both Erez et al. and Ostroff and Harrison discuss limitations of interpretation and analysis that can arise when only level 2 data are available.


251

As illustrated by our extended analysis, the interpretation of the cross-level interaction can vary depending on how the predictor and criterion variables are standardized. Hence, depending on how the variables are standardized, the cross-level interaction hypothesis will address different questions. These different questions may have different answers. Therefore, just as in our extended analysis, there may not be a single answer to the question of whether a cross-level interaction exists in a given set of data. Within the HLM framework, person/organization congruence can be operationalized as a squared Euclidean distance model with an implicit cross-level interaction. Thus the squared Euclidean distance model for person/organization congruence can be examined by testing a hierarchy of quadratic HLM models that contain cross-level interaction terms. By appropriate specification of an HLM model, a variety of research questions can be cast as hypotheses about cross-level interactions. HLM can be used to test such hypotheses in a way that addresses the nesting of persons within organizations, the heterogeneity of regression, and the dependencies of errors within organizations. HLM is readily applied to designs that enhance statistical power by using a large number of organizations. Appendix A The model in Equation 10 can be generalized to cover congruence on several dimensions designated by the subscript k: Yij = β0j* + Σkβkj(Xijk – Wjk)2 + rij, βkj < 0 for all k. Then squaring the terms in the sum on the right side and letting k = 2k gives 2 + Σkβk′j Xijk + rij Yij = β0j + Σkβkj X ijk

(A1)

βkj < 0 for all k, β0j = β0j* + Σkβkj Wjk2 + u0j,

(A2)

βk′j = –2βkjWjk + uk′j for all k.

(A3)

and

Translating these into level 1 and level 2 equations analogous to Equations 1, 2a and 2b yields 2 + Σkβk′j Xijk + rij Yij = β0j + Σkβkj X ijk

(A4)

β0j = γ00 + Σkγ0k Wjk2 + u0j,

(A5)

βkj = γk0 + ukj

(A6)

βk′j = γk′0 + γk′1Wjk + uk′j.

(A7)

and

where γk′0 = 0, and γk′1 = –2βkj.

252


Notes 1. In this special case, the within-organization and between-organization slopes are equal. See Bryk and Raudenbush (1992, pp. 117-123) for an explanation of within-organization and between-organization regression slopes. 2. Viewing this design from an analysis of variance perspective poses two issues regarding variable X. First, in what follows, we treat X as a fixed factor, in effect, assuming that all the possible scores (levels) of X are present in the data. Second, we assume that X is crossed with organizations. The factors may not be crossed in the sample if every value of X does not occur in every organization. This can be viewed as a randomly missing data problem because every value of X is assumed to occur in every organization with some nonzero probability. These two assumptions about X are not critical to our main points: MMR is cumbersome with large numbers of organizations and applying MMR with two level data is not so simple and straightforward as it might at first seem. 3. Equation 6 assumes that the main effect of X is linear and that the X × O interaction is linear. 4. In the special case when the equal prediction model holds, then β1j can be interpreted as the within-group slope even with grand-mean centering.

References Aguinis, H. (1995). Statistical power problems with moderated multiple regression in management research. Journal of Management, 21, 1141-1158. Aguinis, H., Pierce, C. A., & Stone-Romero, E. F. (1994). Estimating the power to detect dichotomous moderators with moderated multiple regression. Educational and Psychological Measurement, 54, 690-692. Aguinis, H., & Stone-Romero, E. G. (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82, 192-206. Aiken, L. S., & West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage. Bryk, A. S., Lee, V. E., & Smith, M. B. (1990). High school organization and its effects on teachers and students: An interpretative summary of the research. In W. H. Clune & J. F. Witte (Eds), Choice and control in American education: Vol. 1. The theory of choice and control in education (pp. 135-226). New York: Falmer Press. Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage. Cronbach, L. J., & Gleser, G. (1965). Psychological tests and personnel decisions (2nd ed.). Urbana: University of Illinois Press. Davison, M. L. (1976). Fitting and testing Carroll’s weighted unfolding model for preferences, Psychometrika, 41, 233-248. Davison, M. L., & Jones, L. E. (1976). A similarity-attraction model for predicting sociometric choice from perceived group structure. Journal of Personality and Social Psychology, 33, 601-612. de Leeuw, J., & Kreft, I. G. G. (1995). Questioning multilevel models. Journal of Educational and Behavioral Statistics, 20(2), 171-189. Edwards, J. (1995). Alternatives to difference scores as dependent variables in the study of congruence in organizational research. Organizational Behavior and Human Decision Processes, 64, 307-324. Erez, A., Bloom, M., & Wells, M. (1996). Using random rather than fixed effects models in meta-analysis: Implications for situational specificity and validity generalization. Personnel Psychology, 49, 275-306.


253

Ghiselli, E. E., Campbell, J. P., & Zedeck, S. (1981). Measurement theory for the behavioral sciences. San Francisco: Freeman. Griffin, M. A. (1997). Interaction between individuals and situations: Using HLM procedures to estimate reciprocal relationships. Journal of Management, 23(6), 759-773. Haberfeld, Y., Semyonov, M., & Addi, A. (1998). A hierarchical linear model for estimating gender-based earnings differentials. Work and Occupations, 25(1), 97-112. Hofmann, D. A. (1997). An overview of the logic and rationale of hierarchical linear models. Journal of Management, 23(6), 723-744. Hofmann, D. A., & Gavin, M. B. (1998). Centering decisions in hierarchical linear models: Implications for research in organizations. Journal of Management, 24(5), 623-641. James, L. R., Joyce, W. F., & Slocum, J. W. (1988). Organizations do not cognize. Academy of Management Review, 13(1), 129-132. Kidwell, R. E., Mossholder, K. W., & Bennett, N. (1997). Cohesiveness and organizational citizenship behavior: A multilevel analysis using work groups and individuals. Journal of Management, 23(6), 775-793. Kreft, I.G.G. (2000). Using random coefficient linear models for the analysis of hierarchically nested data. In H. E. A. Tinsey & S. D. Brown (Eds.), Handbook of Applied Multivariate Statistics and Mathematical Models (pp. 613-639). New York: Academic Press. Kreft, I.G.G., & de Leeuw, J. (1994). The gender gap in earnings: A two-way nested multiple regression analysis with random effects. Sociological Methods and Research, 22(3), 319-341. Kreft, I.G.G., de Leeuw, J., & Aiken, L. S. (1995). The effect of different forms of centering in hierarchical linear models. Multivariate Behavioral Research, 30(1), 1-21. Lee, V. E., & Smith, J. B. (1997). High school size: Which works best and for whom? Educational Evaluation and Policy Analysis, 19, 205-227. Maxwell, S. E., & Delaney, H. D. (1990). Designing experiments and analyzing data. Belmont, CA: Wadsworth. Ostroff, C., & Harrison, D. A. (1999). Meta-analysis: Level of analysis and best estimates of population correlations: Cautions for interpreting meta-analytic results in organizational behavior. Journal of Applied Psychology, 84, 260-270. Pervin, L. A. (1989). Persons, situations, interactions: The history of a controversy and a discussion of theoretical models. Academy of Management Review, 14(3), 350-360. Raudenbush, S. W., Bryk, A. S., Cheong, Y. F., & Congdon, R. T., Jr. (2000). HLM5: Hierarchical linear and nonlinear modeling. Lincolnwood, IL: Scientific Software International. SAS Institute (1996). SAS/STAT Software: Changes and enhancements through release 6.11. Cary, NC: Author. Schmidt, F. L., & Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540. Schmidt, F. L., & Hunter, J. E. (1984). A within-setting empirical test of the situational specificity hypothesis in personnel selection. Personnel Psychology, 37, 317-326. Singer, J. D. (1998). Using SAS PROC MIXED to fit multilevel models, hierarchical models, and individual growth models. Journal of Educational and Behavioral Statistics, 24, 323-355. Vancouver, J. B., Millsap, R. E., & Peters, P. A. (1994). Multilevel analysis of organizational goal congruence. Journal of Applied Psychology, 79, 666-679. Vancouver, J. B., & Schmitt, N. W. (1991). An exploratory examination of person-organization fit: Organizational goal congruence. Personnel Psychology, 44, 333-352. Mark L. Davison (http://www.education.umn.edu/EdPsych/Faculty/Davison.html) is Professor of Educational Psychology and Director of the Office of Educational Accountability at the University of Minnesota. He received a Ph.D. in Quantitative Psychology at the University of Illinois at Urbana-Champaign. His specializations are educational measurement and school accountability.

254


Nohoon Kwak is a research associate in the Office of Educational Accountability at the University of Minnesota. He received a Ph.D. in Educational Psychology from the University of Minnesota, Twin Cities. His current research interests include estimation of interaction effects, centering effects, repeated measures and school accountability. Young Seok Seo is a doctoral candidate in educational psychology at the University of Minnesota. He received an M.A. in Educational Psychology at the University of Minnesota, Twin Cities. His research interests include multicultural counseling, expression of emotions in counseling, and research method. Jiyoung Choi is a doctoral student in educational psychology at the University of Minnesota. She received an M.A. in Educational Psychology at the Seoul National University at Seoul, Korea. Her research interests include cooperative learning, moral psychology, school bullying, and research method.