Overcoming Problems in Confirmatory Factor

This article was downloaded by: [Australian Catholic University] On: 11 May 2014, At: 17:34 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Multivariate Behavioral Research Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hmbr20

Overcoming Problems in Confirmatory Factor Analyses of MTMM Data: The Correlated Uniqueness Model and Factorial Invariance Herbert W. Marsh , Barbara M. Byrne & Rhonda Craven Published online: 10 Jun 2010.

To cite this article: Herbert W. Marsh , Barbara M. Byrne & Rhonda Craven (1992) Overcoming Problems in Confirmatory Factor Analyses of MTMM Data: The Correlated Uniqueness Model and Factorial Invariance, Multivariate Behavioral Research, 27:4, 489-507, DOI: 10.1207/s15327906mbr2704_1 To link to this article: http://dx.doi.org/10.1207/s15327906mbr2704_1

PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or

indirectly in connection with, in relation to or arising out of the use of the Content.

Downloaded by [Australian Catholic University] at 17:34 11 May 2014

This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Multivariate Behavioral Research, 27 (4), 489-507 Copyright O 1992, Lawrence Erlbaum Associates, Inc.

Overcoming Problems in Confirmatory Factor Analyses of MTMM Data: The Correlated Uniqueness Model and Factorial Invariance Herbert W. Marsh Downloaded by [Australian Catholic University] at 17:34 11 May 2014

University of Western Sydney, Macarthur, Australia

Barbara M. Byrne University of Ottawa, Canada

Rhonda Craven. University of New South Wales, Australia

The general model typically used in the confirmatory factor analysis (CFA) approach to multitrait-multimethod (MTMM) data is plagued with methodological problems and frequently results in improper or unstable solutions. Here we reanalyze data from a previously published study, demonstrating that this model may lead to inappropriate interpretations even when it does converge to a proper solution, and describe safeguards against this occurrence. The results support the correlated uniqueness model, diagnostic tests of the validity of CFA-MTMM solutions, the inclusion of external validity criteria in the MTMM design as described by Marsh (1988; 1989; Marsh & Bailey, 19911, and the application of factorial invariance to test the stability of CFA-MTMM solutions. More generally, we demonstrate the flexibility of the CFA-MTMM approach for testing a variety of construct validity issues.

In multivariate behavioral research the interplay between data, statistical models, and substantive interpretation is very complex. Substantive interpretations are likely to suffer if the data, the statistical model, or the match between the two is inappropriate. The use of mathematically elegant, "state of the art7'statistical modelsprovides no guarantee that the substantive conclusions resulting from their application are valid. Because these problems are rarely clearcut, researchers must critically evaluate the validity of substantive interpretations and their implications. These dangers are clearly demonstrated in the confirmatory factor analysis (CFA) approach to multitrait-multimethod (MTMM) data that is the focus of the present investigation. Campbell and Fiske's (1959) MTMM paradigm, despite its continued popularity, is plagued Correspondence should be sent to Herbert W. Marsh, Professor of Education, University of Western Sydney, Macarthur, PO Box 555, Campbelltown NSW 2560 Australia. MULTIVARIATE BEHAVIORAL RESEARCH

489


H. Marsh, B. Byrne, and R. Craven

by problems associated with definitions of terms, operationalizations of criteria, and analytic procedures. Kenny and Kashy (1992) claimed that 30 years after the MTMM paradigm was proposed, we still do not know how to evaluate adequately MTMM data. Whereas a review of these problems is beyond the scopeofthis article, Marsh (1988; 1989)recommended apreliminary inspection of the MTMM matrix using, for example, the Campbell-Fiske guidelines, followed by an evaluation of parameter estimates from the four models in Figure 1 in relation to each other and a priori predictions. Of particular relevance is the CTCU model positing correlated trait factors (CT) and methods as correlated uniquenesses (CU). A growing body of research demonstrates that this model typically results in proper solutions whereas the other models - particularly the widely used CTCM model positing correlated trait factors and correlated method factors - does not. More recently, Kenny and Kashy, lamenting that the rich detail of the general CTCM model is not a realistically achievable goal, argued that it is necessary to introduce simplifying conditions to achieve generally interpretable results, concluding that "as does Marsh (1989), we recommend the Correlated Uniqueness Model for MTMM analysis" (p. 171). Byrne (1989a) used the general CTCM model to compare the multidimensional structure of self-concept of high school students in highability and low-ability tracks. She evaluated relations between four selfconcept traits (TI = General, T2 = School, T3 = English, and T4 = Math) measured by different instruments (MI, M2, and M3). Also available were school performance measures in English, mathematics, and overall grade point average (GPA). Her study was strong in terms of the theoretical basis for a priori predictions about relations among the multiple self-concept traits, the psychometric properties of the different self-concept instruments, the large sample size, and the availability of external validity criteria. Like most studies, Byrne relied primarily on the traditional interpretations of the CTCM model. Somewhat surprisingly, given the tendency for this model to result in improper solutions, the CTCM model converged to proper solutions for both groups. Noting large differences in the parameter estimates for the two groups, Byrne concluded that the structure of self-concept differed in high- and low-ability groups, calling into question the comparability of self-concept responses in different ability groups and the generalizability of self-concept traits measured by the instruments in her study. In the present investigation, a detailed reanalysis of this data incorporating subsequent developments in the CFA approach to MTMM data, we conclude that there is reasonable support for the invariance of the MTMM solutions across the high- and low-ability groups. The juxtaposition of the two sets of conclusions calls into question the

490

MULTIVARIATE BEHAVIORAL RESEARCH



CFA-CTCM

c

CFA-CTUM

CFA-CTCU

u

CFA-CT

c

Figure 1 Four Confirmatory Factor Analysis (CFA) Models For a design with four traits (T) aind four methods (M). Note. Each of the T x M = 16 measured variables (TIMI, T2M1, ... T4M4) is represented by a single measured variable (the boxes) and the latent trait factors (TI, ..., T4) and latent method factors (MI, ... M4) are represented as ovals. CT = correlateid trait model. CTCU = correlated traitjcorrelated uniqueness model. CTUM = correlated trait/uncorrelated method model. CTCM = correlated traitlcorrelated method model. For T > 3, CT is nested under the remaining 3 models, and CTUM is nested under CTCM and CTCW. For T = 3 CTUM and CTCU are equivalent. There is no nesting relation for CTCM and CTCU.


491


H. Marsh, 8.Byrne, and R. Craven

usefulness of the CTCM leading us to ask: "What went wrong and how can we avoid similar problems in the future?" A critical evaluation of specific aspects of the original study suggests that some of Marsh's (1989; Marsh & Bailey, 1991) subsequent warnings about interpretations of the CTCM solution may be warranted. 1. Byrne, commenting on an inspection of the modification indices, indicated that the inclusion of additional correlated uniquenesses would substantially improve the fit. This suggests, perhaps, a need to fit the correlated uniqueness model. 2. Application of the Campbell-Fiske guidelines to the MTMM matrices (see Appendix 1) suggests excellent support for convergent and discriminant validity in both ability tracks. For the high-ability and low-ability tracks respectively, convergent validities are substantial (means rs of .71 and .62) and consistently higher than heterotrait-heteromethod correlations (mean rs of .31 and .30)and heterotrait-monomethod correlations (mean rs .26 and .25). An inspection of the pattern of correlations among different traits also supports a priori predictions. Although differences are small, these preliminary analyses suggest that construct validity may be better in the high-ability track. In marked contrast, the CFA-CTCM interpretations suggested reasonable support for construct validity for the low-ability track, but not for the high-ability track. Because the preliminary inspection of the data conflicted with interpretations based on the more sophisticated CTCM model, Byrne rejected the conclusions based on a formative inspection of the MTMM matrices. Marsh (1989) warned, however, that if there are differences in interpretations based on the Campbell-Fiske guidelines and the CFA solutions, then both sets of interpretations should be interpreted cautiously. 3. The CTCM model resulted in proper solutions for both groups, but there were several uniquenesses that did not differ significantly from zero and parameter estimates with large standard errors. Marsh (1989) suggested that this may indicate an unstable solution and problems in the interpretation of the results even when the model does converge to a proper solution. 4. In the high-track solution, the nonsignificant correlation between Math and School self-concept and particularly the large negative correlation (-.46) between Math and English self-concept are not plausible and violate a priori predictions, calling into question the traditional interpretation of this model. 5. In the high-track solution, the three academic self-concept traits all loaded substantially on each of the method factors and the method factors were very highly correlated. For this pattern of results Marsh (1989) suggested that the so-called method effects may reflect a general trait effect and correlations among these factors may reflect convergence on this general trait across the different methods. 492




Methods Data come from the previously published MTMM study (Byrne, :1989a, 1989b) where the background is presented in greater detail (also see Appendix 1). LISREL VI (Joreskog & Sorbom, 1985) was used to derive maximum likelihood estimates for covariance matrices for the low- and high-ability tracks. Our approach to the evaluation of fit (see Bentler, 1990; Cudeck & Browne, 1983; Marsh, Balla & McDonald, 1988; McDonald & Marsh, 1990) is to: (a) demonstrate that the solution is well defined by establishing that the iterative estimation procedure converges, parameter estimates are within the range of permissible values, and the sizes of the standard errors are reasonable; (b) examine the parameter estimates for alternative models in relation to each other, substantive, a priori expectations, and common sense; (c) evaluate the x2 and subjective indices of fit and compare values obtained frorn alternative models. Historically, the x2 test has been emphasized in CFA studies, but researchers have come to realize the limitations of formal tests of statistical significance. Not only is the test based on restrictive assumptions that are typically unmet, but the intent of a model is to provide an approximation to reality rather than to fully represent all the complexity in the observed data (Cudeck & Browne, 1983). Because any restrictive model with positive df is unable to fit real data and will be rejected by a formal test of significance for a sufficiently large N, more emphasis is placed on subjective indices like the Tucker-Lewis index (TLI; Tucker &Lewis, 1973) and the relative noncentrality index (RNI; McDonald & Marsh) used here. For these indices, values greater than .90 are typically interpreted as indicating an "acceptable" fit. The TLI and RNI differ in that the TLI incorporates a penalty based on the number of estimated paramete:rs whereas the RNI does not (see McDonald & Marsh, for further discussion).

Results Behavior of Models with No Invariance Constraints All four models presented in Figure 1resulted in proper solutions when fit separately for each group (see Table 1,next page). For both tracks, however, the CTCM model failed to converge when LISREL VI was provided with several apparently plausible sets of starting values, but did converge to proper solutions for both tracks when the final solutions presented by Byrne (1 989a) and the startingvalues presented by Byrne (1989b, p. 103) were used. We:refer to this as "start value dependent" in Table 1 and note that this very unusual behavior provides an initial warning that the CTCM solutions may be unstable MULTIVARIATE BEHAVIORAL RESEARCH

493


and need to be interpreted cautiously - a caution that previous research suggests in generally appropriate for CTCM solutions.


Low-track Results For the low-track results, trait-factor loadings and trait-factor correlations are reasonably consistent across the CTCM, CTUM, CT, and CTCU models (see Table 2). The CTCU model fits modestly better than the other models, whereas the CT mode1 fits modestly poorer (see Table 1). Consistent with earlier evaluations of the MTMM matrix and a priori predictions, School selfconcept (T2)is substantially correlated with each of the other self-concept scales whereas Math and English self-concepts are nearly uncorrelated. The CTCU solution differs from the CTCM and CTUM solutions in that the former has all 12 uniquenesses significantly different from zero whereas the latter two each have three uniquenesses that do not differ significantly from zero. On balance, the CTCU model may be preferable to the CTCM model, but the differences are not substantial. Consistent with earlier applications of the Campbell-Fiske criteria, these results indicate very strong trait effects, good discriminant validity, and small method effects. The broad convergence of results based on a priori predictions, application of the Campbell-Fiske guidelines, and the different CFA models provides additional support for these interpretations. High-track Results For the high-track results, trait-factor loadings and trait-factor correlations are reasonably consistent across the CTUM, CT, and CTCU models, but not the CTCM model (see Table 2). The fit of the CT model is modestly poorer than the other models, whereas the fit for the CTCM model is modestly better (see Table 1). The CT, CTUM, and CTCU solutions are generally consistent with a priori predictions, interpretations based on the Campbell-Fiske criteria, and the low-track results. For the CTCM model, however, School self-concept is not substantially correlated with either English or Math self-concept, whereas Math andEnglish self-concepts are substantially negatively correlated (r = -.458) to each other. Also, the results for the CTCM model suggest that the method effects are much larger than those for the other models. The CTCM parameter estimates are dubious and dictate caution in interpreting these results. An inspection of the method factor loadings for the CTCM model supports an alternative explanation proposed by Marsh (1989). For each method factor, the three academic self-concept scales have substantial loadings whereas the 494


H. Marsh, 8.Byrne, and R;. Craven

Table 1 Goodness of Fit and Behavior of Four Models (see F i ~ u r e1): Separate Tests for High-Ability and Low-Abilitv Tracks and Tests of Invariance of Factor Loadincs (FJ& Factor Correlations (FC!. or All Parameters (Total) Across High- and JBW-Ability Tracks

..

Model

-

Invariance Constraint

TLI RNI -

Behavior


CT

Lo Only Hi Only None

FL FL, FC Total

Well defined Well defined Well defined Well defined Well defined Well defined

CTUM

Lo Only Hi Only None FL FL,FC Total

Well defined Well defined Well defined Well defined Improper Solution Improper Solution

CTCM

Lo Only Hi Only

acu

None FL FL, FC Total

Lo Only Hi Only None

FL FL, FC Total

Start Value Dependent Start Value Dependent Start Value Dependent Improper solution Improper solution Improper solution Well defined Well defined Well defined Well defined Well defined Well defined

Note. TLI = Tucker Lewis Index. RNI = Relative Noncentrality Index. For each rnodlel with no invariance constraints (labeled "none" in the invariance constraint column), the X' and df for the multi-sample test is merely the sum of the ~ 3 and s dfi for solutions in each group considered separately (labeled "Lo Only" or "Hi Only"). For each multi-sample test we also defined x2and dffor the multi-sample null model to be the sum of values from the null ]models for each separate group.


495

P (0


m

Table 2

Low Track CTCM" Scale

Trait

Method Unique

2

tlml t2ml t3ml t4ml tlm2 t2m2 t3m2 t4m2 tlm3 t2m3 t3m3 t4m3

337 .756 .751 .728 .648 .724 .753 328 311 .604 .555 .659

.065* -.325 .398 -.064* .449 .405 .115* .046* -.014* .678 .237 .I89

5

Trait Factor Correlations

C r

-I

5g j;

4 m

s n n

F n

0

I

T1 1' 2' T3 T4

.593 .326 .343

CTCU

CTUM

-

.718 .521

Trait

Method Unique

Trait

Unique

Trait

iz


C

I-

.! High