PLS marker variable approach to diagnosing and controlling for method variance Completed Research Paper
Mikko Rönkkö Aalto University, School of Science PO Box 15500, FI-00076 Aalto
[email protected]
Jukka Ylitalo Aalto University, School of Science PO Box 15500, FI-00076 Aalto
[email protected]
Abstract Partial least squares (PLS) path modeling has been adopted as part of the statistical toolbox of many information systems (IS) scholars, particularly when dealing with survey data. Since these data are susceptible to common method variance, several statistical approaches for diagnosing and controlling for this undesirable feature have been developed. While most of these statistical techniques are only applicable to structural equation modeling (SEM), Liang, Saraf, Hu, and Xue (2007) proposed how one of these techniques can be used with PLS analysis. Since this was the first time that a method for controlling common method variance had been made available for PLS users, the method of Liang et al. quickly gained popularity in IS journals. However, recent analysis on the Liang et all approach shows that the method does neither detect nor control for common method variance. In this paper, we propose an alternative PLS marker variable approach for analyzing data contaminated with method variance and provide simulation evidence for the validity of this new approach. Keywords: partial least squares, common method variance, Monte Carlo simulation
Introduction Partial least squares (PLS) path modeling has seen increased use among information systems (IS) researchers as an alternative to structural equation modeling (SEM). Since PLS estimates a composite variable model instead of a system of simultaneous structural equations, analysis approaches developed for SEM are not directly applicable with PLS. Consequently, we have recently seen contributions on how various SEM techniques can be adapted to or implemented with PLS (e.g., Chin, Marcolin, and Newsted 2003; Qureshi and Compeau 2009). A recent paper by Liang, Saraf, Hu, and Xue (2007) joined this group by presenting how the method factor design (cf., P. M. Podsakoff, MacKenzie, Lee, and N. P. Podsakoff 2003) for controlling for common method variance in SEM can be used with PLS. This method was quickly embraced by the IS community to such an extent that it has become almost a standard for papers presenting PLS analyses of survey data in the top journals (e.g., Agarwal and C. Smith 2010; Herath and Rao 2009; Iacovou, Thompson, and H. J. Smith 2009). However, recent analysis concluded that the Liang et al approach does neither detect nor control for common method variance (Chin, Thatcher, and Wright fortcoming). In this paper, we present an alternative PLS-based approach for controlling for common method variance that is similar to the single measured method factor design (Podsakoff, MacKenzie, Jeong-Yeon Lee, & Podsakoff, 2003) or CFA marker variable design in SEM (Richardson, Simmering, and Sturman 2009). We test the effectiveness of the new PLS marker variable approach using a simulation study that provides support for the new approach. The method and the simulation are focused explicitly on reflective indicators, since formative indicators are assumed to be error-free (Bollen, 1989: 222-223), and thus incompatible with data that can contain common method variance.
Thirty Second International Conference on Information Systems, Shanghai 2011
1
Research Methods and Philosophy
Controlling for Common Method Variance Common method bias is a subset of method bias (Burton-Jones 2009) that arises in quantitative research when the measured relationship between two constructs is either inflated or attenuated compared to the true value as a result of covariance caused by the measurement approach, rather than the measured trait. The presence of common method variance in the data affects both the estimates of construct validity and the estimates of relationships between the constructs in the study. Since the correlated method variance component increases the overall correlation between the items, it increases the estimates of convergent validity of measurement scales and reduces the estimates of discriminant validity between different scales in a way that is difficult to detect with standard tests of construct validity (Richardson et al. 2009; Straub, Boudreau, and Gefen 2004). Common method bias is also manifested in the relationships between constructs most typically increasing the estimates of the relationships between constructs (Richardson et al. 2009). While minimizing the effects of common method variance can be done on multiple levels, starting from the study design and data collection, our paper focuses only on statistical remedies available after the data have been collected. Podsakoff et al. (2003) divide these methods into the Harman's single-factor test, partial correlation procedures, various method factor designs, the correlated uniqueness model, and the direct product model. The most commonly used method for accessing the extent of method variance is the Harman's single-factor test (cf. Podsakoff & Organ, 1986). In this technique, all indicator variables in the model are analyzed using an exploratory common factor analysis, and the extent to which the common method variance might be a problem is determined by examining the first factor of the unrotated factor solution. The problem with this approach is that it is a very insensitive test and the lack of one dominant factor does not mean that the data are not contaminated with method variance (P. M. Podsakoff et al. 2003), and consequently this test should be avoided if other alternatives can be used (Malhotra, Kim, and Patil 2006; P. M. Podsakoff et al. 2003). Moreover, the method is often misused by substituting the common factor analysis with a component analysis (e.g. Pavlou and El Sawy 2006). The second class of analytical methods used to detect and control for common method variance is partial correlation methods. These involve partialling out a measured method effect (e.g. a social desirability score), a marker variable, or a general factor score. The partialling procedure can be performed either by partialling a proxy for method variance from the observed data or by calculating a correlation matrix from the data and then partialling out a marker correlation from all other correlations. In these approaches the detection of method variance is accomplished by comparing the results of the analyses before and after the partialling procedure. A few IS papers have used a variant of partialling method variance score from the data by using the first principal component of all the indicators as a control variable in each inner model regression (Dong, Xu, and Zhu 2009; Pavlou and El Sawy 2006; Pavlou, Liang, and Xue 2007). A key weakness of this approach is that the extracted principal component can also contain covariance that is attributable to the substantive effect between the constructs (Kemery and Dunlap 1986) causing underestimation of the model parameters. Another partial correlation procedure that has been adopted by IS researchers involves partialling out a marker correlation from the construct correlation matrix (Lindell and Whitney 2001). If a marker variable that has been defined a priori is used, the correlation between this variable and all other variables is assumed to result solely from method variance and can be partialled out from other correlations in the study. Although this approach can detect common method bias and provides a correction for the regression coefficients, it has been criticized as having several conceptual and empirical problems (P. M. Podsakoff et al. 2003), that its effectiveness has not yet been thoroughly tested (Lance, Dawson, Birkelbach, and Hoffman 2010), or that it can falsely detect common method bias where none exists in the data (Richardson et al. 2009). Moreover, there is one particular feature of this test that makes it incompatible with PLS analysis: the t-test of the statistical significance of a correlation is a parametric test and thus requires the expected sampling distribution of the correlations between the constructs to be known. However, the sampling distribution of the correlation of PLS scores does not necessarily follow any known sampling distribution (Dijkstra 1983), making this an inappropriate method for testing for method variance in PLS studies1. 1
A proof is available from the first author by request.
2
Thirty Second International Conference on Information Systems, Shanghai 2011
PLS marker variable approach
The third class of methods are the so-called method factor designs. These approaches, which are designed for SEM, explicitly model method effects in the data (e.g., Williams and Anderson 1994). The idea is to include one or more method factors into the analysis to model the systematic measurement error caused by using the same measurement method for multiple items. Technically, these methods rely on partialling out the error variance on the indicator level so that it does not affect the parameter estimates in the structural model. While these approaches can be very useful in SEM (Richardson et al. 2009), they are not directly applicable to PLS because of differences in model estimation. Although in the measurement theory and in the model specification stage of PLS the constructs can be considered as latent variables, they are not estimated as such (Marcoulides, Chin, and Saunders 2009; Reinartz, Haenlein, and Henseler 2009) but are approximated as linear combinations of the indicators. If these indicators contain measurement error, the composite variables will also be contaminated (Bollen and Lennox 1991). Additionally, the single latent method factor design requires the method factor to be set to be uncorrelated with all study constructs, but this kind of constraint is not possible in a PLS model. Finally, the last two models presented in the paper by Podsakoff et al. (2003) – the correlated uniqueness model and the direct product model - are also explicitly designed for SEM. We do not present these methods in detail since we are not aware of applications of these two models in IS research. While the SEM approaches presented above are not directly applicable to PLS, as explained above, there has been at least one attempt to develop a PLS-based approach for detecting and controlling for method variance: Liang, Saraf, Hu, and Xue (2007) proposed an adaptation of the single unmeasured method factor design for PLS. Their approach is to create a single indicator construct for each indicator and treat these as endogenous constructs determined by the constructs of the model being tested and a method factor that are all modeled as second-order reflective constructs. The problem is that although this model would be equivalent to the single unmeasured method factor design if estimated as a SEM model, using it with PLS rests on the incorrect assumption that PLS would estimate a SEM model. The composite variable model used by PLS cannot provide error free construct scores because indicator variance is not partialled, but all variance is included in the construct scores. Subsequently, a recent simulation study concluded that the Liang et al approach can neither detect nor control for common method variance (Chin et al. fortcoming).
A PLS marker variable Method for Controlling for Common Method Variance As no valid PLS-based approach for detecting and controlling for common method variance currently exists, we propose a new “PLS marker variable” modeling approach. Since controlling for common method variance completely during the outer estimation is not possible with PLS due to the fact that the construct scores will always include at least the same amount of method variance as the least contaminated indicator, our approach to controlling for method variance is to do it during the inner estimation. We propose that a method factor be included as a predictor for all endogenous constructs in the model. This means that we accept that the estimates of constructs are affected by measurement error and try to minimize the consequences of this fact for the path coefficient estimates. Our approach differs from the principal component approach by using a directly measured method factor rather than building a proxy based on the substantive indicators in the model, since including the first principal component of all indicators is known to cause bias in the results (Kemery and Dunlap 1986). Contrary to the SEM-based CFA marker approaches, the method factor should not load on the indicators of the study constructs due to the fact that a construct loading on the items of another independent variable would be collinear and shared items with the dependent variable would cause the coefficients between the method factor and endogenous constructs to be inflated. This would cause severe bias in the estimates of the coefficients between the model constructs2. More specifically, we propose a six-step procedure for detecting and controlling for common method variance when using PLS path modeling. We present all the steps below and explain them in more detail later in the paper. In the following discussion we use the term ‘baseline model’ for the original PLS model and ‘method factor model’ for a model where controls have been added for method variance. 2
We also tested this approach, but the results were not useful and hence are not included in the paper.
Thirty Second International Conference on Information Systems, Shanghai 2011
3
Research Methods and Philosophy
1.
Choose marker indicators.
2. Diagnose the presence of common method variance in the data. 3. Analyze the data with the baseline PLS model without any controls for method variance. 4. Add a new construct to the PLS model estimated in step 3 using the indicators identified in step 1. Use this construct as a predictor for all endogenous constructs. 5.
Analyze the results from step 2 and compare the results from steps 3 and 4. If common method variance seems to cause bias in the results, report the path coefficients from step 4 as the main results.
6. If the results from step 4 are reported as the main results, calculate R2 values for each inner model regression and the fit statistics depending on these statistics manually. Also calculate corrected values for construct validity tests and standardized factor loadings. The advantage of this approach is that apart from the initial diagnostics and calculating the adjusted construct validity and model fit statistics, the entire procedure can be implemented with any PLS software. We will now explain each of the six steps in more detail. Step 1. Since estimating a PLS marker variable model requires several marker indicators to be included in the model, the first step in the analysis should be identifying a set of marker indicators. These can be deliberately included marker variable items or items that were collected in the same survey but are not included in the model being tested. The items should be as close as possible to ideal marker variables (Williams, Hartman, and Cavazotte 2010), which means that they should not correlate with the study variables apart from the correlation caused by common method variance. We suggest that the indicators are chosen on the basis of the correlation matrix of all the items collected in the survey. An ideal candidate for a marker variable should be minimally correlated with the indicators of the study variables (Lindell and Whitney 2001), but must be subject to the same measurement effects as the study constructs (P. M. Podsakoff et al. 2003; Williams et al. 2010). In other words, if most of the constructs are measured with attitude scales, e.g., a question about work experience in years would not qualify as a good marker variable. Since the overall quality of the marker variables used in IS research has been poor (Sharma, Yetton, and Crawford 2009), a researcher seeking good marker items should venture to review papers outside the IS discipline (e.g., Williams et al. 2010). Indicators of general constructs such as social desirability or general affectivity can be used (P. M. Podsakoff et al. 2003). Some IS researchers have also had success using unrelated attitude scales, such as fantasizing (Son and Kim 2008) or trusting stance (Li, Hess, and Valacich 2008) For the PLS marker variable model, the marker indicators can be individual items, partial or complete scales, or any combination of these. The factor structure of the marker items is not important in itself, as long as the constructs that the marker indicators measure are approximately uncorrelated with the study constructs. However, the correlations between the study items and the marker items need to be positive and approximately equal after the sampling error of these correlations has been taken into account. If a correlation between one study item and one marker item is significantly higher than other correlations between study items and marker items, then the relationship between these two indicators cannot be explained solely by method variance and the marker indicator should be considered for removal. The quality of a method factor that is estimated as a component extracted from a set of items depends on the number of items used to form this composite because the typical effect sizes that the method variance source is assumed to have on the items is small compared to factor loadings expected for normal constructs. If only a small part of variance in the marker items is caused by the measurement method, then the number of indicators required for reliable measurement of method variance increases. Additionally, increasing the number of indicators for the method factor decreases the biasing effects that any spurious correlation between the marker indicators and study indicators might have. We cover the issue of the number of marker items in more detail later in the paper, but at a minimum the number of marker items should be equal to the number of indicators in the endogenous construct with the most indicators. Step 2. The analysis of data where common method variance can potentially cause problems should always start by diagnosing the degree of common method variance in the data. We recommend using the
4
Thirty Second International Conference on Information Systems, Shanghai 2011
PLS marker variable approach
mean correlation between the marker items identified in the previous step and the study items as the primary test for method variance. Since these two groups of indicators should be related to each other only because they are affected by a shared method factor, applying the tracing rules of path analysis (cf., Bollen 1989) shows that in this case the mean loading on the method factor for these items is the square root of the mean correlation. If the mean correlation is close to zero, method variance is most probably not an issue with the data. We propose a rule of thumb of .05 which would mean that 5% of the variance in the data is method variance. As a secondary option, we also recommend using the partial correlation procedure described by Lindell and Whitney (2001). Although it should not be used for the correlations estimated based on PLS construct scores because of violations in the distributional assumptions, it can be used for construct score estimates calculated with simple summed scales. Another alternative is to analyze the data with SEM using the CFA marker technique (Williams et al. 2010). Step 3. After the initial diagnostics, the baseline model should be estimated without including any controls for common method variance. Step 4. In the fourth step a method construct measured by all items identified in the first step is included in the model as an exogenous variable predicting each endogenous construct. This new construct does not need to pass the normal tests for construct validity (e.g. Composite Reliability, AVE, unidimensionality). If the indicators are related only because they share the same source of method variance, this effect is typically not strong enough to make the composite formed from these items pass the tests for scale reliability. If the indicators are not unrelated, but come from one or more scales measuring constructs that should be unrelated to the study constructs, the composite can pass the tests for reliability and validity. However, since the reliable variance caused by a construct that is unrelated to the study constructs does not affect the inner model regressions, the reliability and validity statistics are not relevant for the method factor model. Step 5. After analyzing this model with marker variable design, it should be compared with the baseline model estimated in step 3. If the regression paths that were significant in step 3 are not significant in step 4, this is an indication that the data have a method variance problem. On the basis of this comparison and the diagnostics for common method variance calculated in step 2, the researcher should make a decision whether to report the results where common method variance is controlled for or report the results of the baseline model. If there is sufficient evidence to consider common method variance a problem in the data, leading to a decision to use the controlled model as the main results, the sixth step in the procedure must be performed. Step 6. In the sixth step we calculate corrected values for R2 statistics and standardized factor loadings. The reason for this is that although the inner model paths are controlled for common method variance, the construct values are not and the inner model R2 statistics also include the effect of the method factor in addition to the effects of the study constructs. The R2 statistics are adjusted by exporting the construct values from the PLS analysis and then calculating fitted values for each inner model regression using the path coefficients from the PLS estimation but excluding the method factor. The corrected R2 values are calculated as the square of the correlations between the construct scores and the fitted values. Standardized factor loadings and indicators of the validity of the construct measurement can be obtained from a correlation matrix of indicators and construct values from PLS. Because the observed correlation between the variables is a sum of the correlation caused by the method variance and the substantive variance (cf., Lindell and Whitney 2001), we can calculate a corrected correlation matrix by subtracting the method variance correlation calculated during step 2 from the observed correlation matrix. Since a standardized regression coefficient of a bivariate regression equals the correlation of the variables, we can obtain the standardized factor loadings directly from this adjusted correlation matrix. After this calculating the most commonly used measures for construct validity is relatively straightforward because these (e.g. factor analysis and average variance extracted) are based on the correlations between indicators and between the indicators and constructs. If a choice must be made, the item to construct cross-loading matrix should be reported using the corrected correlation matrix, but it is preferable that both the corrected and raw correlations are included in order to give the reader more information on which to assess what impact common method variance and the procedure used to control for this have on the study.
Thirty Second International Conference on Information Systems, Shanghai 2011
5
Research Methods and Philosophy
Monte Carlo Study of Common Method Variance Since it is difficult to know if common method variance is present in the data, any useful method should not only control for the possible bias but would also remove only a negligible amount of variance if the data were truly not contaminated with method variance (Richardson et al. 2009). We tested the suggested PLS marker variable approach for its ability to control for method variance where it exists in the data and the sensitivity of the method in cases where the data are not contaminated with method variance using Monte Carlo simulation. A similar approach has previously been used when testing new PLS modeling approaches (Chin et al. 2003; Qureshi and Compeau 2009) and also when testing the effectiveness of different approaches for controlling for common method variance (Richardson et al. 2009) in SEM. In practice this means generating multiple samples of data from a known population and testing the performance of a statistical test with these data. The first step in a Monte Carlo analysis is choosing a population model from which the Monte Carlo samples are generated. Because of its prevalence in IS and frequency of use in CMV studies (Malhotra et al. 2006; Sharma et al. 2009), we chose to focus on the technology acceptance model (TAM). Since the original TAM model is fairly simple, we chose to use the extended TAM model as presented in a metaanalysis paper by Schepers and Wetzels (2007) as the population model. We set the number of indicators in the model to three for all the six latent variables, as it is a fairly common number in IS research (Chin et al. 2003). The tested model is shown in Figure 1. When generating the Monte Carlo samples, common method variance was modeled by including a method factor on which all the indicator variables had a loading and setting this factor to be uncorrelated with all the other latent variables. The loadings were constrained to be equal so that the latent method factor produced equal, systematic measurement error in all indicators. In addition, we created a set of marker variables that were uncorrelated with all the other indicators in the model except for the correlation caused by the method factor. The standardized factor loadings between the indicators and the model constructs were set to 0.7, as this value is recommended as a minimum standard in the IS literature and was also used in a prior Monte Carlo study by Chin, Marcolin, and Newsted (2003). To test the applicability of our proposed method with different types of data, we included several experimental factor based on what has been used in earlier similar studies (Chin et al. 2003; Richardson et al. 2009). The first factor was to vary the level of common method variance in the population. We chose four levels of none, low, moderate, and high and set the factor loadings on the method factor as 0, 0.1, 0.3, and 0.5, respectively. The second factor was the sample size for which we chose six different values: 25, 50, 100, 200, 500, and 1000. The third factor was the number of marker variables, since the operationalization of the method factor is typically of interest in method factor studies (Richardson et al. 2009) and because the accuracy of PLS estimates is dependent on the number of indicators (Hui and Wold 1982). We used four levels of method factor indicators: 3, 6, 9, and 18. Additionally, we included a fifth model with no controls added for method variance to see how much common method variance would bias the parameter estimates of an uncontrolled model. In total, the combination of these three factors resulted in a full factorial design of 120 (4 x 6 x 5) unique modeling conditions. We selected the number of replications for each condition to be 500, which is a commonly used number in Monte Carlo simulation studies (e.g. Chin et al., 2003). The Monte Carlo samples were generated with Mplus 6.1 structural equation modeling software and the PLS models were estimated with plspm-package (version 0.1-11) of the R statistical software environment. The number of marker variables was varied in the model according to the data and when the data did not include marker variables, we estimated the model without a method factor.
6
Thirty Second International Conference on Information Systems, Shanghai 2011
PLS marker variable approach
Figure 3: Tested model
Figure 1. Tested model
Results We start the results section by presenting how the baseline model, where we did not include a method factor, performed under varying levels of method variance. Table 1 shows the mean estimate of each path coefficient for six different sample sizes and four different levels of method variance. We did not include the population values in the table since it is well known that PLS typically produces path estimates that are attenuated (Chin 1998; Dijkstra 1983), but compare the estimates with PLS estimates obtained when the baseline model is estimated with data free of method variance. Table 1. Path coefficients when method factor is not included Table 1: Path coefficients when method factor is not included Sample size and method variance 25 0 BIU->ASU PEU->ATU PU->ATU SN->ATU ATU->BIU PEU->BIU PU->BIU SN->BIU PEU->PU SN->PU
0.1 0.3
50 0.5
0
100 0.5
0
0.1 0.3
0.45 0.43 0.48 0.55 0.42 0.42 0.46 0.52 0.40 0.40 0.20 0.19 0.21 0.26 0.18 0.18 0.21 0.24 0.18 0.19 Figure Population model of 0.32 six constructs and a0.35 method0.31 factor 0.31 0.32 4:0.31 0.33 0.35 0.32 0.32 0.07 0.07 0.11 0.14 0.06 0.08 0.10 0.14 0.07 0.08 0.17 0.16 0.18 0.19 0.17 0.16 0.18 0.20 0.17 0.16 0.08 0.11 0.11 0.15 0.10 0.11 0.11 0.14 0.09 0.10 0.25 0.25 0.27 0.27 0.25 0.25 0.27 0.28 0.25 0.26 0.13 0.11 0.16 0.15 0.12 0.11 0.14 0.15 0.11 0.12 0.29 0.30 0.36 0.38 0.29 0.29 0.33 0.37 0.27 0.28 0.20 0.20 0.23 0.29 0.20 0.20 0.23 0.29 0.19 0.19 200 0
BIU->ASU PEU->ATU PU->ATU SN->ATU ATU->BIU PEU->BIU PU->BIU SN->BIU PEU->PU SN->PU
0.1 0.3
0.39 0.17 0.31 0.07 0.17 0.09 0.25 0.11 0.27 0.18
0.1 0.3 0.39 0.18 0.31 0.07 0.16 0.10 0.26 0.11 0.28 0.18
0.44 0.21 0.32 0.10 0.18 0.11 0.27 0.13 0.31 0.22
500 0.5
0
0.51 0.24 0.34 0.13 0.19 0.14 0.28 0.16 0.36 0.27
0.38 0.17 0.31 0.07 0.17 0.10 0.25 0.11 0.26 0.17
0.1 0.3 0.39 0.18 0.31 0.07 0.17 0.10 0.26 0.11 0.27 0.18
0.43 0.20 0.32 0.10 0.18 0.12 0.27 0.13 0.31 0.22
0.45 0.21 0.33 0.10 0.18 0.11 0.26 0.13 0.32 0.22
0.5 0.51 0.24 0.34 0.14 0.19 0.14 0.28 0.15 0.37 0.27
1000 0.5
0
0.50 0.24 0.34 0.13 0.19 0.14 0.28 0.15 0.36 0.27
0.38 0.17 0.31 0.06 0.17 0.09 0.25 0.10 0.26 0.17
0.1 0.3 0.39 0.18 0.31 0.07 0.17 0.10 0.26 0.11 0.27 0.18
0.43 0.20 0.32 0.10 0.18 0.12 0.26 0.13 0.31 0.22
0.5 0.50 0.24 0.34 0.13 0.19 0.14 0.28 0.15 0.36 0.27
Mean of parameter estimate over 500 Monte Carlo samples.
The first observation from the table is that method variance generally increases the estimates of the path coefficients. This increase does not seem to be proportional to the path estimate without method variance or the population value, but depends on the number of constructs in each inner model regression. The
Thirty Second International Conference on Information Systems, Shanghai 2011
7
Research Methods and Philosophy
reason for this is that since in this simulation the method variance affects all constructs equally, it increases collinearity in the regression model and hence causes suppression effects in the model. With low levels of method variance the suppression effect can result in some path estimates actually decreasing when method variance increases. The path coefficients seem to decrease when the sample size increases. This is a counterintuitive feature since it means that less data would provide stronger evidence of the presence of a path in the model. The effect is caused by the combination of the fact that sampling distribution of error correlations is wider for smaller sample sizes and the fact that PLS considers all variance as useful variance leading to the inflation of path coefficients in the presence of error correlations in the data (Rönkkö and Ylitalo 2010). When the sample size decreases, the expected values of absolute values of error correlations increase, thus increasing the estimates of path coefficients. Apart from this effect, there do not seem to be any strong effects of sample size on the inflation of path coefficients caused by method variance. As a conclusion, the PLS estimates seem to be affected only a little if the method variance is low (0.1), but on the moderate (0.3) and high (0.5) levels of method variance, the bias is substantial and needs to be controlled for. Next we tested the ability of the proposed approach to control for method variance. Since the purpose of controlling for method variance is to obtain parameter estimates that are close to the parameter estimates if the same model was estimated with data that are free of method variance, we used the model without the method factor estimated with data without method variance as the reference model. If the PLS marker variable approach is a useful tool for controlling for method variance, the parameter estimates should be closer to the reference model values than the biased values shown in Table 1. Table 2. Path coefficients with marker variable model for data with high method variance Table 3: Path coefficients with marker variable model for data with high method variance Sample size and method indicators 25 Ref BIU->ASU Method->ASU PEU->ATU PU->ATU SN->ATU Method->ATU ATU->BIU PEU->BIU PU->BIU SN->BIU Method->BIU PEU->PU SN->PU Method->PU
0
3
0.45 0.55 0.48 0.17 0.20 0.26 0.23 0.32 0.35 0.34 0.07 0.14 0.12 0.10 0.17 0.19 0.19 0.08 0.15 0.14 0.25 0.27 0.26 0.13 0.15 0.13 0.09 0.29 0.38 0.33 0.20 0.29 0.25 0.15
50 6
9
18
0.44 0.22 0.21 0.32 0.10 0.14 0.17 0.13 0.25 0.12 0.14 0.30 0.23 0.22
0.42 0.25 0.20 0.30 0.09 0.18 0.16 0.12 0.24 0.11 0.17 0.28 0.21 0.25
0.38 0.30 0.19 0.28 0.07 0.23 0.15 0.11 0.23 0.10 0.23 0.25 0.18 0.32
Ref
0
3
0.42 0.52 0.46 0.17 0.18 0.24 0.22 0.32 0.35 0.34 0.06 0.14 0.12 0.10 0.17 0.20 0.19 0.10 0.14 0.13 0.25 0.28 0.27 0.12 0.15 0.13 0.08 0.29 0.37 0.33 0.20 0.29 0.25 0.13
100 6
9
18
0.43 0.22 0.21 0.32 0.11 0.13 0.18 0.12 0.26 0.13 0.11 0.31 0.23 0.18
0.41 0.24 0.20 0.32 0.10 0.16 0.18 0.11 0.26 0.12 0.14 0.29 0.22 0.21
0.38 0.28 0.19 0.31 0.08 0.19 0.17 0.10 0.25 0.11 0.17 0.27 0.20 0.25
Ref
0
3
0.40 0.51 0.46 0.17 0.18 0.24 0.22 0.31 0.34 0.33 0.07 0.14 0.12 0.09 0.17 0.19 0.18 0.09 0.14 0.12 0.25 0.28 0.27 0.11 0.15 0.14 0.07 0.27 0.37 0.33 0.19 0.27 0.24 0.12
6
9
18
0.44 0.21 0.21 0.32 0.10 0.12 0.18 0.12 0.27 0.13 0.10 0.31 0.22 0.16
0.42 0.23 0.20 0.32 0.10 0.14 0.17 0.11 0.27 0.12 0.12 0.30 0.21 0.19
0.40 0.26 0.19 0.31 0.09 0.16 0.17 0.10 0.26 0.11 0.15 0.28 0.19 0.23
Sample size and method indicators 200 Ref BIU->ASU Method->ASU PEU->ATU PU->ATU SN->ATU Method->ATU ATU->BIU PEU->BIU PU->BIU SN->BIU Method->BIU PEU->PU SN->PU Method->PU
0
3
0.39 0.51 0.46 0.16 0.17 0.24 0.22 0.31 0.34 0.33 0.07 0.13 0.11 0.08 0.17 0.19 0.18 0.09 0.14 0.12 0.25 0.28 0.27 0.11 0.16 0.14 0.07 0.27 0.36 0.33 0.18 0.27 0.24 0.12
500 6
9
18
0.44 0.20 0.21 0.32 0.10 0.11 0.18 0.12 0.27 0.13 0.10 0.31 0.22 0.16
0.42 0.22 0.21 0.32 0.10 0.13 0.18 0.11 0.27 0.13 0.11 0.30 0.21 0.18
0.41 0.24 0.20 0.31 0.09 0.15 0.17 0.10 0.26 0.12 0.13 0.29 0.20 0.21
Ref
0
3
0.38 0.50 0.46 0.16 0.17 0.24 0.22 0.31 0.34 0.33 0.07 0.13 0.11 0.08 0.17 0.19 0.19 0.10 0.14 0.13 0.25 0.28 0.27 0.11 0.15 0.14 0.07 0.26 0.36 0.33 0.17 0.27 0.24 0.12
1000 6
9
18
0.43 0.20 0.21 0.32 0.10 0.11 0.18 0.12 0.27 0.13 0.09 0.32 0.22 0.15
0.42 0.22 0.20 0.32 0.09 0.12 0.18 0.11 0.26 0.13 0.11 0.31 0.21 0.17
0.40 0.24 0.19 0.31 0.08 0.15 0.17 0.11 0.26 0.12 0.13 0.29 0.20 0.20
Ref
0
3
0.38 0.50 0.46 0.16 0.17 0.24 0.22 0.31 0.34 0.33 0.06 0.13 0.11 0.08 0.17 0.19 0.19 0.09 0.14 0.13 0.25 0.28 0.27 0.10 0.15 0.14 0.07 0.26 0.36 0.33 0.17 0.27 0.24 0.12
Mean of parameter estimate over 500 Monte Carlo samples. Reference values (Ref) are PLS estimates without method factor and with no method variance in the data.
8
Thirty Second International Conference on Information Systems, Shanghai 2011
6
9
18
0.43 0.20 0.21 0.33 0.10 0.11 0.18 0.12 0.27 0.13 0.09 0.32 0.22 0.15
0.42 0.21 0.20 0.32 0.10 0.12 0.18 0.12 0.26 0.13 0.10 0.31 0.21 0.17
0.41 0.24 0.19 0.32 0.09 0.15 0.18 0.11 0.26 0.12 0.12 0.29 0.20 0.20
PLS marker variable approach
Tables 2 and 3 show the results of applying the PLS marker variable approach to data with high and moderate levels of method variance. In these cases the parameter estimates of models without controls for method variance are inflated, as shown in Table 1, and hence controlling for method variance is necessary. With 100 or more observations the models with 9 and 18 marker indicators provide path coefficients that are very close to the reference model, but the models with 3 and 6 marker indicators provide only partial correction for common method bias. When there is a sufficient amount of data the PLS marker variable approach is hence a useful approach for analyzing data contaminated with common method variance and generally the number of indicators in the method factor improves the performance of the model. As for the smaller sample sizes, with 50 observations the method factor seems to account for too much of the variance of the dependent variable, attenuating all path coefficients below the reference model when a large number of method indicators are used. Although some path coefficients are estimated close to the reference model values when the number of method indicators is between 3 and 9, the effect is not as consistent as with the larger sample sizes. Although with 25 observations the mean difference between the path estimates of marker variable model and the reference model is similar to the results calculated with the data sets of 50 observations, the effect varies considerably more between paths with the smallest data. Although PLS estimates are generally more consistent when the sample size and number of indicators is large, the sample size also needs to be considerably larger than the number of indicators in the largest block (Hui and Wold 1982). This explains why the models with 9 and 18 method indicators do not behave as well with smaller amounts of data. Table 3. Path coefficients with marker variable model for data with moderate variance Table 4: Path coefficients with marker variable model for data with moderate method variance Sample size and method indicators 25 Ref BIU->ASU Method->ASU PEU->ATU PU->ATU SN->ATU Method->ATU ATU->BIU PEU->BIU PU->BIU SN->BIU Method->BIU PEU->PU SN->PU Method->PU
0
3
0.45 0.48 0.44 0.07 0.20 0.21 0.20 0.32 0.33 0.31 0.07 0.11 0.09 0.05 0.17 0.18 0.18 0.08 0.11 0.10 0.25 0.27 0.26 0.13 0.16 0.15 0.05 0.29 0.36 0.33 0.20 0.23 0.21 0.09
50 6
9
18
0.42 0.11 0.19 0.29 0.09 0.08 0.17 0.10 0.24 0.15 0.09 0.31 0.19 0.12
0.40 0.18 0.18 0.28 0.08 0.13 0.15 0.09 0.23 0.14 0.13 0.29 0.18 0.18
0.34 0.23 0.16 0.25 0.07 0.20 0.12 0.08 0.20 0.12 0.23 0.25 0.15 0.28
Ref
0
3
0.42 0.46 0.44 0.07 0.18 0.21 0.20 0.32 0.32 0.31 0.06 0.10 0.09 0.06 0.17 0.18 0.18 0.10 0.11 0.11 0.25 0.27 0.26 0.12 0.14 0.13 0.06 0.29 0.33 0.31 0.20 0.23 0.21 0.09
100 6
9
18
0.42 0.12 0.20 0.30 0.09 0.09 0.17 0.10 0.25 0.13 0.10 0.29 0.20 0.13
0.40 0.16 0.19 0.29 0.08 0.12 0.16 0.09 0.24 0.12 0.14 0.28 0.19 0.18
0.37 0.21 0.17 0.27 0.07 0.18 0.15 0.08 0.23 0.11 0.19 0.25 0.17 0.26
Ref
0
3
0.40 0.45 0.43 0.08 0.18 0.21 0.20 0.31 0.33 0.32 0.07 0.10 0.09 0.06 0.17 0.18 0.18 0.09 0.11 0.11 0.25 0.26 0.26 0.11 0.13 0.13 0.05 0.27 0.32 0.31 0.19 0.22 0.20 0.08
6
9
18
0.42 0.11 0.20 0.31 0.09 0.09 0.17 0.10 0.25 0.12 0.08 0.30 0.19 0.13
0.41 0.14 0.19 0.31 0.08 0.11 0.17 0.10 0.25 0.12 0.11 0.29 0.19 0.16
0.39 0.18 0.18 0.29 0.07 0.15 0.16 0.09 0.24 0.11 0.14 0.26 0.17 0.21
6
9
18
0.42 0.10 0.19 0.32 0.09 0.06 0.18 0.11 0.26 0.12 0.05 0.29 0.20 0.09
0.41 0.12 0.19 0.32 0.08 0.07 0.18 0.11 0.26 0.12 0.06 0.29 0.20 0.10
0.40 0.14 0.18 0.31 0.08 0.09 0.17 0.10 0.26 0.11 0.07 0.28 0.19 0.12
Sample size and method indicators 200 Ref BIU->ASU Method->ASU PEU->ATU PU->ATU SN->ATU Method->ATU ATU->BIU PEU->BIU PU->BIU SN->BIU Method->BIU PEU->PU SN->PU Method->PU
0
3
0.39 0.44 0.43 0.08 0.17 0.21 0.20 0.31 0.32 0.32 0.07 0.10 0.09 0.05 0.17 0.18 0.18 0.09 0.11 0.11 0.25 0.27 0.26 0.11 0.13 0.12 0.04 0.27 0.31 0.30 0.18 0.22 0.21 0.08
500 6
9
18
0.42 0.11 0.20 0.31 0.09 0.08 0.18 0.11 0.26 0.12 0.07 0.30 0.20 0.11
0.41 0.13 0.19 0.31 0.08 0.09 0.17 0.10 0.26 0.12 0.08 0.29 0.19 0.13
0.40 0.16 0.19 0.30 0.08 0.12 0.17 0.10 0.25 0.11 0.10 0.28 0.18 0.16
Ref
0
3
0.38 0.43 0.42 0.08 0.17 0.20 0.20 0.31 0.32 0.32 0.07 0.10 0.09 0.05 0.17 0.18 0.18 0.10 0.12 0.11 0.25 0.27 0.26 0.11 0.13 0.12 0.04 0.26 0.31 0.30 0.17 0.22 0.21 0.07
1000 6
9
18
0.42 0.10 0.19 0.32 0.09 0.07 0.18 0.11 0.26 0.12 0.06 0.29 0.20 0.09
0.41 0.12 0.19 0.31 0.08 0.08 0.18 0.11 0.26 0.12 0.07 0.29 0.20 0.11
0.40 0.15 0.18 0.31 0.08 0.10 0.18 0.10 0.26 0.11 0.08 0.28 0.19 0.13
Ref
0
3
0.38 0.43 0.42 0.08 0.17 0.20 0.20 0.31 0.32 0.32 0.06 0.10 0.09 0.05 0.17 0.18 0.18 0.09 0.12 0.11 0.25 0.26 0.26 0.10 0.13 0.12 0.04 0.26 0.31 0.30 0.17 0.22 0.21 0.06
Mean of parameter estimate over 500 Monte Carlo samples. Reference values (Ref) are PLS estimates without method factor and with no method variance in the data.
Table 4 and Table 5 show the mean difference between path coefficients obtained with the marker variable model and the reference model when there is only low method variance or actually no method
Thirty Second International Conference on Information Systems, Shanghai 2011
9
Research Methods and Philosophy
variance in the data. As Table 1 shows, in these cases it is not really necessary to introduce any controls for method variance in the model, since the potential bias is non-existent or negligible. However, it can be difficult to know if this is the case and hence we need to know the potential downside if the PLS marker variable approach is applied when there is no method variance problem in the data. Table 4 and Table 5 show that, if no common method variance is present, adding a control for it tends to bias the results downwards. This finding is in line with previous studies (Richardson et al. 2009) and the key question is whether this bias is sufficient to cause the interpretation of the results to change. For the largest sample sizes (200 and more) there seems to be practically no change in the path coefficients, regardless of the number of indicators used. For a sample size of 100 the changes in path coefficients become larger and can potentially affect the significance of the parameter estimates for paths where the population effects are small. For the smallest sample sizes the path coefficients generally become biased if a method factor is included but the data do not have a method variance problem. Table 4. Path coefficients with marker variable model for data with low method variance Table 5: Path coefficients with marker variable model for data with low method variance Sample size and method indicators 25 Ref BIU->ASU Method->ASU PEU->ATU PU->ATU SN->ATU Method->ATU ATU->BIU PEU->BIU PU->BIU SN->BIU Method->BIU PEU->PU SN->PU Method->PU
0
3
0.45 0.43 0.41 -0.00 0.20 0.19 0.18 0.32 0.31 0.31 0.07 0.07 0.08 -0.02 0.17 0.16 0.16 0.08 0.11 0.11 0.25 0.25 0.24 0.13 0.11 0.11 -0.00 0.29 0.30 0.29 0.20 0.20 0.19 -0.00
50 6
9
18
0.39 -0.04 0.18 0.28 0.08 0.03 0.16 0.11 0.22 0.11 -0.02 0.28 0.18 0.00
0.37 -0.00 0.18 0.26 0.08 0.02 0.14 0.10 0.21 0.10 -0.01 0.26 0.17 0.00
0.31 0.02 0.16 0.21 0.06 0.01 0.10 0.10 0.17 0.10 0.00 0.23 0.15 0.02
Ref
0
3
0.42 0.42 0.41 0.01 0.18 0.18 0.18 0.32 0.32 0.31 0.06 0.08 0.08 0.00 0.17 0.16 0.16 0.10 0.11 0.11 0.25 0.25 0.25 0.12 0.11 0.11 0.01 0.29 0.29 0.28 0.20 0.20 0.19 0.01
100 6
9
18
0.39 0.01 0.17 0.30 0.07 -0.00 0.15 0.11 0.24 0.11 0.01 0.27 0.19 0.00
0.38 0.03 0.17 0.29 0.07 0.02 0.14 0.10 0.23 0.10 0.03 0.26 0.18 0.03
0.34 0.03 0.16 0.26 0.07 0.05 0.12 0.10 0.20 0.10 0.03 0.24 0.16 0.04
Ref
0
3
0.40 0.40 0.40 0.01 0.18 0.19 0.19 0.31 0.31 0.31 0.07 0.08 0.08 0.01 0.17 0.16 0.16 0.09 0.10 0.10 0.25 0.26 0.26 0.11 0.12 0.12 -0.00 0.27 0.28 0.28 0.19 0.19 0.19 0.01
6
9
18
0.39 0.01 0.18 0.30 0.08 0.02 0.16 0.10 0.25 0.12 0.01 0.27 0.19 0.01
0.38 0.02 0.18 0.30 0.08 0.02 0.15 0.10 0.25 0.11 0.02 0.27 0.18 0.03
0.36 0.04 0.18 0.28 0.07 0.03 0.14 0.10 0.23 0.11 0.03 0.25 0.17 0.04
Sample size and method indicators 200 Ref BIU->ASU Method->ASU PEU->ATU PU->ATU SN->ATU Method->ATU ATU->BIU PEU->BIU PU->BIU SN->BIU Method->BIU PEU->PU SN->PU Method->PU
0
3
0.39 0.39 0.39 0.02 0.17 0.18 0.18 0.31 0.31 0.31 0.07 0.07 0.07 0.01 0.17 0.16 0.16 0.09 0.10 0.10 0.25 0.26 0.26 0.11 0.11 0.11 0.01 0.27 0.28 0.28 0.18 0.18 0.18 0.01
500 6
9
18
0.39 0.01 0.18 0.31 0.07 0.02 0.16 0.10 0.26 0.11 0.01 0.27 0.18 0.03
0.39 0.02 0.18 0.30 0.07 0.02 0.16 0.10 0.25 0.11 0.03 0.27 0.18 0.04
0.38 0.04 0.18 0.29 0.07 0.05 0.15 0.09 0.25 0.11 0.05 0.26 0.17 0.07
Ref
0
3
0.38 0.39 0.39 0.01 0.17 0.18 0.18 0.31 0.31 0.31 0.07 0.07 0.07 0.01 0.17 0.17 0.17 0.10 0.10 0.10 0.25 0.26 0.26 0.11 0.11 0.11 0.00 0.26 0.27 0.27 0.17 0.18 0.18 0.01
1000 6
9
18
0.39 0.01 0.18 0.31 0.07 0.01 0.17 0.10 0.26 0.11 0.01 0.27 0.18 0.02
0.39 0.03 0.18 0.31 0.07 0.03 0.17 0.10 0.25 0.11 0.03 0.27 0.18 0.04
0.38 0.04 0.17 0.30 0.07 0.05 0.16 0.10 0.25 0.11 0.04 0.26 0.18 0.07
Ref
0
3
0.38 0.39 0.39 0.01 0.17 0.18 0.18 0.31 0.31 0.31 0.06 0.07 0.07 0.01 0.17 0.17 0.17 0.09 0.10 0.10 0.25 0.26 0.26 0.10 0.11 0.11 0.01 0.26 0.27 0.27 0.17 0.18 0.18 0.01
6
9
18
0.39 0.02 0.18 0.31 0.07 0.01 0.17 0.10 0.26 0.11 0.02 0.27 0.18 0.02
0.39 0.03 0.18 0.31 0.07 0.02 0.17 0.10 0.26 0.11 0.03 0.27 0.18 0.04
0.39 0.04 0.17 0.31 0.07 0.04 0.17 0.10 0.25 0.10 0.04 0.26 0.18 0.06
Mean of parameter estimate over 500 Monte Carlo samples. Reference values (Ref) are PLS estimates without method factor and with no method variance in the data.
While the differences in mean parameter estimates describe the accuracy of the PLS marker variable approach in recovering the original path coefficients, this is not a sufficient condition to conclude that the PLS marker variable approach is a valid method for analyzing data with common method variance issues. To check if the parameter estimates are stable, we also inspected the standard deviations (not reported) of the path estimates over all replications in each experimental condition for each path. These analyses indicated that the common method variance artificially reduces the standard deviation of the parameter estimates, but after the method factor is applied, the standard deviations of parameter estimates increase to their original level or close to this level.
10 Thirty Second International Conference on Information Systems, Shanghai 2011
PLS marker variable approach
Table 5. Path coefficients with marker variable model for data with no method variance Table 6: Path coefficients with marker variable model for data without method variance Sample size and method indicators 25 0 BIU->ASU Method->ASU PEU->ATU PU->ATU SN->ATU Method->ATU ATU->BIU PEU->BIU PU->BIU SN->BIU Method->BIU PEU->PU SN->PU Method->PU
3
0.45 0.42 0.00 0.20 0.19 0.32 0.31 0.07 0.06 0.01 0.17 0.17 0.08 0.08 0.25 0.24 0.13 0.12 -0.01 0.29 0.27 0.20 0.19 0.02
50
6
9
18
0.39 -0.01 0.19 0.30 0.06 -0.01 0.16 0.08 0.23 0.12 0.01 0.26 0.18 0.01
0.37 -0.01 0.18 0.28 0.05 0.01 0.15 0.07 0.22 0.11 -0.00 0.24 0.18 0.02
0.31 0.02 0.17 0.23 0.04 0.02 0.11 0.07 0.17 0.10 0.00 0.21 0.15 -0.00
BIU->ASU Method->ASU PEU->ATU PU->ATU SN->ATU Method->ATU ATU->BIU PEU->BIU PU->BIU SN->BIU Method->BIU PEU->PU SN->PU Method->PU
3
0.39 0.39 -0.00 0.17 0.17 0.31 0.31 0.07 0.07 0.00 0.17 0.17 0.09 0.09 0.25 0.25 0.11 0.11 -0.01 0.27 0.27 0.18 0.18 -0.00
3
0.42 0.40 -0.00 0.18 0.18 0.32 0.32 0.06 0.06 0.01 0.17 0.17 0.10 0.09 0.25 0.25 0.12 0.12 0.00 0.29 0.29 0.20 0.20 0.01
100
6
9
18
0.39 -0.00 0.17 0.31 0.06 0.00 0.16 0.09 0.24 0.12 -0.01 0.28 0.19 -0.01
0.38 -0.00 0.17 0.30 0.06 0.01 0.15 0.09 0.23 0.11 0.01 0.27 0.18 0.01
0.35 0.00 0.16 0.27 0.05 0.00 0.13 0.09 0.20 0.11 0.00 0.24 0.17 0.02
0
3
0.40 0.39 -0.00 0.18 0.17 0.31 0.31 0.07 0.07 0.00 0.17 0.17 0.09 0.09 0.25 0.25 0.11 0.11 0.00 0.27 0.27 0.19 0.19 0.00
Sample size and method indicators 500
200 0
0
6
9
18
0.38 -0.01 0.17 0.31 0.06 0.00 0.17 0.09 0.25 0.11 -0.00 0.26 0.18 -0.00
0.38 -0.00 0.17 0.30 0.06 0.00 0.17 0.09 0.25 0.11 -0.01 0.26 0.18 -0.01
0.37 0.00 0.17 0.29 0.06 0.00 0.16 0.09 0.24 0.11 -0.02 0.25 0.17 -0.01
0
3
0.38 0.38 -0.00 0.17 0.17 0.31 0.31 0.07 0.07 -0.00 0.17 0.17 0.10 0.10 0.25 0.25 0.11 0.11 -0.00 0.26 0.26 0.17 0.17 -0.00
6
9
18
0.38 -0.00 0.17 0.31 0.07 -0.00 0.17 0.10 0.25 0.10 -0.00 0.26 0.17 -0.00
0.38 -0.01 0.17 0.31 0.07 0.00 0.17 0.10 0.25 0.10 -0.00 0.26 0.17 0.00
0.38 -0.00 0.17 0.30 0.07 -0.00 0.17 0.09 0.25 0.10 0.00 0.26 0.17 -0.00
0
6
9
18
0.39 -0.00 0.17 0.31 0.07 0.00 0.16 0.09 0.25 0.11 0.00 0.26 0.18 0.01
0.38 0.00 0.17 0.30 0.07 0.01 0.16 0.09 0.24 0.11 -0.00 0.26 0.18 -0.01
0.36 -0.01 0.17 0.28 0.06 0.01 0.15 0.09 0.22 0.11 0.01 0.24 0.17 0.01
1000 3
0.38 0.38 -0.00 0.17 0.17 0.31 0.31 0.06 0.06 0.00 0.17 0.17 0.09 0.09 0.25 0.25 0.10 0.10 -0.00 0.26 0.26 0.17 0.17 -0.00
6
9
18
0.38 -0.00 0.17 0.31 0.06 0.00 0.17 0.09 0.25 0.10 -0.01 0.26 0.17 -0.00
0.38 0.00 0.17 0.31 0.06 -0.00 0.17 0.09 0.25 0.10 -0.01 0.26 0.17 -0.00
0.38 0.00 0.17 0.31 0.06 -0.00 0.17 0.09 0.25 0.10 -0.00 0.26 0.17 0.00
Mean of parameter estimate over 500 Monte Carlo samples.
Example We will next present an example of using the proposed approach. Since the example requires data that are known to be contaminated with method variance, weonpicked one of our Monte Carlo samples having a Table 7: Factor loadings method factor moderate method variance problem. InMethod this variance case we to use 75 observations and nine marker anddecided method indicators indicators and use this to testNone the extended Weak TAM(0.1) model shown in Figure 1. High (0.5) Moderate (0.3) Step 1. First weSample need to3 choose Although defined marker 6 9 the18marker 3 6indicators. 9 18 3 6 9 this 18 data 3 has 6 99 a priori 18 indicators, we need that appropriate markers. done the indicator 25.00 to check 0.26 0.11 0.09these 0.05 are 0.28 0.11 0.11 0.06 0.39 0.27 0.27This 0.24 is0.59 0.51by 0.48examining 0.45 0.26 0.10 in 0.08Table 0.04 6. 0.27 0.11are 0.11interested 0.06 0.43 0.33 0.30 correlations 0.28 0.63 0.54 between 0.51 0.47 marker items and level covariance 50.00 matrix shown We in the 100.00 are0.28 0.09 0.08in 0.03the0.25 0.11 0.11 0.06 of 0.48columns 0.38 0.36 0.32 0.66 0.56 0.52and 0.49 rows M1-M9. The study items, which located cross-section PEU1-ASU3 200.00 0.25 0.08 0.08 0.03 0.26 0.11 0.11 0.06 0.55 0.43 0.39 0.34 0.67 0.57 0.53 0.49 correlations seem to vary between -0.22 and 0.37 most being positive. Since the width of the 99% 500.00 0.24 0.09 0.08 0.03 0.28 0.12 0.12 0.08 0.60 0.47 0.41 0.36 0.68 0.57 0.54 0.49 confidence interval for 0.25 sampling of Pearson’s correlation coefficient for sample size 75 is 1000.00 0.07 0.07 distribution 0.03 0.30 0.16 0.15 0.11 0.61 0.47 0.42 0.36 0.68 0.58 0.54 0.49 approximately 0.59, can conclude that the can be estimates for the same method variance Mean ofwe factor loadings over all indicators and correlations 500 Monte Carlo samples. correlation. In other words, there is no need to believe that any marker item would have truly stronger association with the study items. Step 2. The second step is to estimate the method variance correlation. This is done by taking a mean of the correlations inspected in the previous step. The mean correlation between study items and marker items is 0.074 This is larger than our rule of thumb of 0.05 and indicates that we should run both the baseline model and the PLS marker variable model and compare the results. At this point we do not do
Thirty Second International Conference on Information Systems, Shanghai 2011
11
PEU1 PEU2 PEU3 SN1 SN2 SN3 PU1 PU2 PU3 ATU1 ATU2 ATU3 BIU1 BIU2 BIU3 ASU1 ASU2 ASU3 M1 M2 M3 M4 M5 M6 M7 M8 M9
0.68 0.73 0.69 0.03 -0.09 -0.08 -0.00 -0.04 -0.09 0.54 0.10 0.19 0.05 0.49 0.53 0.41 0.40 0.36 0.24 0.16 0.30 0.34 0.45 0.35 0.01 -0.05 0.16 0.62 0.32 0.33 0.34 0.09 0.02 0.21 0.67 0.63 0.33 0.35 0.49 0.03 -0.01 0.08 0.29 0.44 0.39 0.28 0.32 0.32 0.05 0.11 0.16 0.26 0.28 0.22 0.59 0.40 0.43 0.54 -0.04 0.11 0.10 0.26 0.32 0.40 0.67 0.20 0.34 0.35 0.25 0.06 0.21 0.34 0.29 0.35 0.38 0.18 0.23 0.27 0.10 0.00 0.17 0.25 0.27 0.35 0.40 0.36 0.29 0.43 0.15 -0.06 0.20 0.25 0.22 0.29 0.42 0.14 0.23 0.16 0.16 0.09 0.27 0.23 0.11 0.21 0.12 0.37 0.32 0.30 0.17 0.12 0.24 0.36 0.21 0.27 0.15 0.27 0.34 0.27 0.14 -0.06 0.12 0.28 0.28 0.22 0.18 0.22 0.05 0.17 0.09 0.03 0.03 0.23 0.30 0.27 0.26 0.06 0.05 -0.12 -0.01 -0.06 0.06 0.08 0.11 -0.01 0.04 0.37 0.27 0.29 -0.11 -0.11 -0.05 0.02 0.24 0.00 0.24 -0.02 0.12 0.10 0.02 -0.01 0.22 0.05 0.02 -0.03 0.16 0.05 0.08 0.00 0.04 0.17 -0.04 0.00 0.21 -0.22 0.09 0.21 0.30 0.25 -0.08 0.10 0.04 0.18 0.15 0.05 0.05 0.03 0.19 -0.00 -0.12 0.02 0.12 0.02 0.08 0.13 -0.01 -0.05 0.06 -0.04 -0.10 -0.12 0.04 0.07 0.21 0.23 -0.00 -0.06 -0.10 0.03 0.03 0.25 0.10 -0.01 0.00 0.13 0.06 0.57 0.41 0.50 0.37 0.42 0.74 0.43 0.49 0.66 0.71 0.10 0.16 0.31 0.24 0.36 0.16 0.10 0.30 0.27 0.39 0.60 0.07 0.17 0.42 0.44 0.48 0.67 0.66 0.32 0.18 0.08 -0.03 0.11 0.13 0.23 0.08 -0.10 0.04 -0.00 0.04 0.23 0.06 0.30 0.11 0.11 0.14 0.21 0.10 0.16 0.19 0.05 0.11 0.01 0.21 0.15 0.07 0.07 0.07 -0.04 -0.08 -0.05 -0.09 0.01 0.10 0.03 0.08 0.01 0.08 0.22 0.14 0.05 0.15 -0.02 -0.10 -0.06 -0.03 -0.10 0.13 -0.01 0.10 0.19 0.09 0.08 0.09 0.07 0.05 0.11 0.11 0.08 0.09 -0.04 0.15 0.15 0.22 -0.00 -0.03 0.14 -0.15 0.18 -0.05
Table 11: Correlations for example PEU1 PEU2 PEU3 SN1 SN2 SN3 PU1 PU2 PU3 ATU1 ATU2 ATU3 BIU1 BIU2 BIU3 ASU1 ASU2 ASU3
0.04 0.18 0.00 0.18 0.20 -0.00 0.18 0.20
M1
0.11 0.19 -0.04 0.16 -0.01 0.17 -0.14
M2
0.08 0.24 0.14 -0.10 0.26 0.08
M3
-0.02 0.28 -0.16 0.16 0.10
M7
M8 M9
0.22 0.11 0.13 0.03 0.27 -0.12 0.15 0.15 0.12 -0.07
M4 M5 M6
Research Methods and Philosophy
any more diagnostics for method variance, but will return to the issue of diagnostics when we have estimated the PLS models. Table 6. Example correlation matrix
12 Thirty Second International Conference on Information Systems, Shanghai 2011
PLS marker variable approach
Step 3. Next we run the PLS model using plspm R package. The results of this analysis are shown the in Table 7. Although all paths were positive in the population model, only six out of ten estimates are statistically significantly different from zero indicating that the 75 observations give us rather low statistical power to estimate this model. Step 4. After running the baseline model, we add a new construct for method factor in the model. This is an exogenous construct that has a regression path to each endogenous construct. We use the nine indicators identified in step 1 as the indicators for this construct. The results of this analysis are shown in Table 7. Table 7. PLS Table 12: PLS pathresults estimates Baseline model Marker variable model Paths BIU->ASU M->ASU PEU->ATU PU->ATU SN->ATU M->ATU ATU->BIU PEU->BIU PU->BIU SN->BIU M->BIU PEU->PU SN->PU M->PU
Est.
S.E
sig.
Est.
S.E
sig.
0.47
0.07
0.000
0.40 0.23 0.02
0.11 0.11 0.11
0.001 0.043 0.390
0.45 0.09 0.11 0.15
0.10 0.10 0.11 0.13
0.000 0.279 0.248 0.209
0.46 0.20
0.10 0.08
0.000 0.023
0.43 0.21 0.37 0.21 0.03 0.13 0.45 0.09 0.11 0.15 -0.01 0.41 0.20 0.17
0.09 0.12 0.10 0.12 0.12 0.12 0.12 0.12 0.12 0.14 0.15 0.10 0.12 0.10
0.000 0.085 0.001 0.082 0.388 0.224 0.001 0.292 0.266 0.223 0.397 0.000 0.102 0.099
R2 ASU ATU BIU PU
0.22 0.31 0.35 0.26
0.26 0.32 0.35 0.29
Goodness of fit Absolute Relative Outer.mod Inner.mod
0.46 0.89 0.99 0.89
0.42 0.81 0.99 0.82
Step 5. Now that we have estimated both the baseline model and the marker variable model, we can compare the results that are shown side by side in Table 7. The first observation is that all parameter estimates that were significant in the baseline model are slightly smaller in the marker variable model. Table 13:variance Cross loading matrices This means that the correction for method has removed path inflation caused by method Marker variable model the SN -> PU Corrected valuesoriginally significant at pBIU ATU wasSN originally at p
