An Evaluation of Construct Validity: What Is This Thing ...

HUMAN PERFORMANCE, 18(4), 445–462 Copyright © 2005, Lawrence Erlbaum Associates, Inc.

An Evaluation of Construct Validity: What Is This Thing Called Emotional Intelligence? David L. Van Rooy Burger King Corporation

Chockalingam Viswesvaran and Paul Pluta Florida International University

This article presents a meta-analytic review of the Emotional Intelligence (EI) construct. The first portion of the study examines the relation between EI measures based on two differing models of the construct (i.e., mixed and ability). This study then examines the relation of each of the models separately with cognitive ability and the Big Five personality factors. Results indicate that measures based on the mixed model of EI overlap extensively (i.e., correlate .71 among themselves; k = 12, N = 3,259), whereas mixed measures and ability measures are relatively distinct (.14; k = 13, N = 2,442). Mixed model measures of EI exhibited greater overlap with personality- than ability-based EI measures. Conversely, ability-based EI measures demonstrated a higher correlation with cognitive ability than mixed measures (.34 vs. .13). Implications and suggestions for the measurement of EI are provided.

The concept of Emotional Intelligence (EI) has received extensive attention in both the scientific and practitioner literature in recent years (Bar-On, 2000; Brackett & Mayer, 2003; Mayer, Salovey, Caruso, & Sitarenios, 2001; Roberts, Zeidner, & Matthews, 2001). Several symposia in conferences, special issues of journals, and edited texts have also focused on this construct. In the popular literature, as well as in empirical studies, claims have been made regarding the criticality of EI for effective leadership, successful learning in classes, and for maintaining positive interpersonal relationships. Many applied areas of psychology have probed the utility of this construct in their respective domains. This coincides with a recent Correspondence should be sent to David Van Rooy, Department of Psychology, Florida International University, Miami, FL 33199. E-mail: [email protected]

446

VAN ROOY, VISWESVARAN, PLUTA

upsurge of interest in noncognitive predictors for personnel selection in industrial-organizational psychology. This is due, in part, to problems (e.g., group differences) that accompany cognitive tests, which are the strongest single predictor of job performance (Hunter & Hunter, 1984; Reilly & Chao, 1982; Schmidt & Hunter, 1998). In particular, the influence of personality has been one of the most extensively studied areas in this realm (e.g., Barrick & Mount, 1991; Hough & Ones, 2001; Hurtz & Donovan, 2000). Personality tests may not mitigate all adverse impact concerns when used as part of a selection battery (Ryan, Ployhart, & Friedel, 1998), but are relatively unbiased and free from many of the concerns attendant with cognitive tests when they are used in isolation. However, although personality measures are desirable from this perspective, their validity is considerably lower than tests of cognitive ability and industrial-organizational psychologists have continued to explore alternative predictors. One of these is EI. Recent studies have shown that EI demonstrates moderate levels of predictive validity (Van Rooy & Viswesvaran, 2004), but there is still considerable debate over what constitutes the construct of EI and clarification in this area is urgently needed. This article discusses a focal measurement concern (i.e., construct validity) associated with EI and then applies meta-analytic techniques to provide a better understanding of its construct validity. More specifically, we examine the link between the two different models of EI that have been postulated as well as the relation of each of the models separately with personality and cognitive ability.

TWO MODELS OF EI EI is an appealing, yet controversial, construct for many reasons. First, as mentioned, studies have shown that it can be a useful predictor of performance in many situations (Van Rooy & Viswesvaran, 2004). It should be noted that although the findings of Van Rooy and Viswesvaran have been used to tout the potential predictive value of EI, it has also been recognized that they found little incremental validity over cognitive ability. There is also a clear disjoint when discussions are raised about what actually constitutes the construct. Generally, there are two main divisions in this debate. The first side claims that EI is nothing more than a renaming of existing constructs. For example, Eysenck (1998, p. 110) stated that psychologists should dismiss those who talk of EI (which, according to Eysenck, is nothing but a combination of intelligence and emotional stability) just as physicists would dismiss any scientist who talks of “hot lengths” as a new concept (which, according to Eysenck, is only a combination of temperature and length). The other side believes that EI is unique, but within this group rests another issue. In other words, proponents of EI generally recognize that there are currently two different models (e.g., Hedlund & Sternberg, 2000; Mayer, Salovey, & Caruso, 1999). One of these is referred to as an ability model and the other as a mixed

EI CONSTRUCT VALIDITY

447

model. As discussed later, the ability model defines EI as a type of intelligence whereas the mixed model is more dispositional or trait-based in nature (Petrides & Furnham, 2001) with less of a cognitive emphasis. Although many psychologists find utility in both models, others are extremely critical of the mixed model. Daus and Ashkanasy (2003), for instance, recently wrote a commentary strongly asserting that they do not endorse the mixed model. They stated that mixed models (e.g., Bar-On, 1997; Goleman, 1995) may have value in certain organizational settings, but are too broad in scope, and do not differ enough from personality and competency models to be described as EI (p. 69). They further stated that the only acceptable method of measuring EI is skill or behavior based and preferably not self-report (i.e., not ability based, in this case). Overlooked in their discussion, however, are problems incumbent with the performance-based scoring formats. With intelligence tests, there is generally assumed to be one correct answer for each question. For instance, there can be little argument that 10 times 10 equals 100. With performance-based EI, the solution is not as definitive. Instead, respondents are accorded points based on the percentage of respondents who endorse a similar response (various scoring methods such as expert and consensus can be used to derive the “correct” answers). Another issue involves discriminating among the most emotionally intelligent people. It is entirely possible that a simple and feasible option may be picked by most people when in actuality a different response could be the best choice. In this case, then, the individuals with the highest EI may not be given the most points and the test will not be able to properly discriminate among respondents. This is an area where research is needed to assess this possibility. Others have similarly described measures derived from the different models as competing (Brackett & Mayer, 2003), but the two may still be more complementary than conflicting given their underlying premise (Ciarrochi, Chan, & Caputi, 2000). Accordingly, although the models are often postulated as separate constructs, it does not necessarily mean that the two are mutually exclusive. Along this line, McCrae (2000) has suggested that it would be more prudent to recognize that the processing of emotional information involves specific abilities and certain personality traits instead of debating whether EI should be classified as a disposition or ability (p. 272). Our results will be valuable for more clearly delineating the potential interchangeability of the constructs or if the two should indeed be considered separately. In fact, if measures based on the two models correlate highly, then there would be no need to assess the differential relation of each with personality and cognitive ability. If the two models truly are distinct, there should be a minimal relation between them. In fact, naming jargon aside, a review of the extant literature provides little theoretical rationale suggesting that the models should demonstrate a large degree of convergence. The mixed model of EI has been described as a conglomeration of traits, dispositions, skills, competencies, and abilities that do not necessarily have a

448

VAN ROOY, VISWESVARAN, PLUTA

strong emotional or intellective component, in spite of the EI name that is attached (Caruso, Mayer, & Salovey, 2002, p. 307). Originators of the mixed model have recognized the multifaceted nature of the model and described it accordingly. For instance, Bar-On (1997) indicated that the term intelligence was used to describe the aggregate collection of skills, competencies, and abilities; emotional was attached as a prefix to point out that this intelligence differed from cognitive ability (p. 15). The self-report Emotional Quotient Inventory (EQ-i; Bar-On, 1997), which is the most extensively used mixed model measure, consists of five general factors: interpersonal EQ, intrapersonal EQ, adaptability, stress management, and general mood. These five factors are further broken into 15 subscales (Bar-On, 2000). Because these subscales measure areas such as empathy, independence, and optimism, the scale undoubtedly taps into certain facets of personality; the current analyses will be of value in determining what are those factors. Just as the mixed model may have more overlap with personality than the ability model, the latter has been postulated to have a stronger cognitive component. Indeed, because the ability model has been postulated as a type of intelligence, a moderate relation should exist between the two constructs. For a construct to be classified as a type of intelligence, it has been suggested that it should conform to three criteria (Mayer, Caruso, & Salovey, 1999). First, the construct must include an actual form of mental behavior. In this case, it would be the ability to recognize emotions and solve emotionally laden problems. Second, the construct should be moderately related to other intelligences. That is, EI should overlap, but not be redundant, with other forms of intelligence (e.g., verbal). Finally, it should be able to be developed and increase with age and experience. Initial studies have shown that EI scores increase with age (Bar-On, 1997; Day & Carroll, 2004; Mayer et al., 1999; Van Rooy, Alonso, & Viswesvaran, 2005), but others have reported nonsignificant differences (Roberts et al., 2001). In should be noted that the latter finding might have been partly attributable to range restriction in the sample. Considering the differential focus of the two models, both called EI, it would be surprising if measures based on the different models were highly correlated. Due to the large samples sizes that are often involved in meta-analyses, the correlation may still be significant, but not as high as would be expected if the two models were tapping into an isomorphic construct. It is important to draw out the distinction between the two models of EI for several reasons. First, if EI is to be established as a legitimate construct, researchers must clearly delineate what the underlying pieces consist of and how the models are related. For instance, although the Big Five are all considered under the personality rubric, each dimension is relatively unique. This departs from EI where the two models are both supposed to measure “emotional intelligence,” yet are most likely measuring disparate individual difference variables. Second, this study examines both of the models in relation to personality and cognitive ability. The true value of EI will ultimately depend on how well it predicts relevant criteria, but this

EI CONSTRUCT VALIDITY

449

will be a moot point if the differential relations (and possibility for incremental validity) with other constructs are not adequately examined separately for the two models to determine if they should be classified as the same construct or as unique. The first wave of EI literature mostly commented on the conceptual differences between the models, but empirical studies are now starting to more closely evaluate this issue. Accordingly, the first portion of this study cumulates all of these studies and provides an overall correlation between the models. The results will provide a much more thorough understanding of the constructs before we turn to how they are related to personality and cognitive ability.

PERSONALITY AND EI Perhaps the most common argument raised by skeptics of EI is that it simply draws from the different facets of personality. As mentioned, it has been suggested that EI is no more than a combination of emotional stability and intelligence (Eysenck, 1998); empirical results are not so supportive of this claim. In a review of the Big Five, McCrae (2000) instead suggested that EI should be most strongly related to the openness to experience dimension, but this claim is also lacking strong empirical evidence. When the various relations that have been proposed between the Big Five and EI are compared to the discrepant results that have been found, it becomes apparent that there is not a clear understanding of how the constructs are intertwined. Moreover, as discussed later, research indicates that the relations often depend on what model the EI measure of interest is based. Given the potential overlap of EI and personality, it is important to assess where the constructs are redundant. Dawda and Hart (2000) administered the self-report EQ-i (Bar-On, 1997) and found particularly high correlations with personality as measured with the NEO-FFI (Costa & McCrae, 1991). In fact, all of the Big Five factors had correlations in excess of .40 with the exception of Openness to Experience. Of the Big Five factors, Openness to Experience has recurrently shown lower correlations with EI, as measured with the mixed model, than the other four factors, but the results are still mixed (cf. Janovics & Christiansen, 2001). Interestingly, this is the opposite of findings for the ability model, which has typically shown higher correlations with Openness to Experience. Other studies using mixed model measures have also found high correlations with the Big Five and the general consensus is that significant overlap exists between the constructs (Davies, Stankov, & Roberts, 1998). As mentioned, this is to be expected because mixed models often measure individual difference variables, such as empathy and optimism, which are included in the personality domain. Still, it remains to be determined if, and to what extent, mixed model tests are measuring other components that are unique from personality. The relation between EI and personality takes on a different shape when an ability-based measure is used. Unlike the mixed model, initial evidence suggests that

450

VAN ROOY, VISWESVARAN, PLUTA

the ability model appears to be rather distinct from existing personality traits. Agreeableness appears to be the only personality factor that consistently shows a moderate relation with EI when measured with a test based on the ability model (e.g., Brackett & Mayer, 2003; Brackett, Mayer, & Warner, 2004; Janovics & Christiansen, 2001); Openness to Experience has also shown moderate correlations with EI but has not been as consistent as Agreeableness and has even been shown to be negatively correlated (–.22; Lopes, Salovey, & Straus, 2003). Although studies have varied somewhat in the magnitude of their findings in this area, it is clear that measures of EI based on the ability model do not share as strong of a relation with personality as do measures based on the mixed model. This differing relation for the two models again suggests that two relatively distinct constructs are actually being measured.

COGNITIVE ABILITY AND EI Research has also demonstrated that the relation between EI and cognitive ability is contingent on what method is used to measure EI. The ability model of EI argues that EI should be considered as a type of intelligence, akin to verbal and spatial intelligence. Accordingly, the two constructs should be moderately, but not highly, correlated. In contrast, the mixed model of EI makes no such assertion and does not claim to be an actual type of intelligence. Thus, measures of EI based on the mixed model of EI have no theoretical basis suggesting that it should be correlated with cognitive ability. Indeed, it has even been suggested that the mixed model measures almost everything other than cognitive ability (Hedlund & Sternberg, 2000). Unlike the overlap of mixed model EI tests and personality, there does not appear to be a heavy cognitive component underlying these measures. Using the EQ-i (Bar-On, 1997), Derksen, Kramer, and Katzo (2002) found a correlation of only .07 between the EI and a test of cognitive ability; EQ-i subscales ranged from –.07 to .12. Newsome, Day, and Catano (2000) found a similar correlation between the EQ-i and a test of cognitive ability (.08). These findings are relatively consistent (e.g., Brackett & Mayer, 2003; Janovics & Christiansen, 2001) and rarely do correlations exceed .10. A notable exception to this was reported in a large-scale study (n > 3,000) by Fund and Bar-On (2004) which found a correlation of .25 between the EQ-i and cognitive ability as measured by the Raven Progressive Matrices. In contrast, tests based on the ability model appear to have a heavier cognitive component and studies have routinely found correlations in excess of .30 (Roberts et al., 2001). It may be that the relation of an ability-based test of EI and cognitive ability depends on what aspect of cognitive ability is measured. For instance, Ciarrochi et al. (2000), using Raven Progressive Matrices, which could be classified as perfor-

EI CONSTRUCT VALIDITY

451

mance IQ instead of verbal, found only a minimal relation between the XXXXX (MEIS) and cognitive ability. Studies that have used measures that could be described as verbal intelligence have typically found higher correlations (e.g., Brackett & Mayer, 2003; Mayer et al., 1999). These types of studies are of particular importance and should continue, as they will help further define EI.

METHOD Search for Primary Data Similar to Van Rooy and Viswesvaran (2004), this meta-analysis only included studies that used a measure that was specifically described as a test of EI. Their meta-analysis included no studies before the year 1995 and this served as the starting point for the current literature search. EI was first conceptualized before this date (Salovey & Mayer, 1990), but all of the known published work during the intervening years is primarily nonempirical in nature. A computer and manual search were both conducted to find any articles that reported a relation between measures of EI or between EI and personality or cognitive ability. The computer portion of the search involved entering key words (e.g., emotional, construct, personality, cognitive, etc.) into the PsychInfo and PsycFirst databases. All articles that appeared to meet the aforementioned criteria were gathered and the reference sections of these articles were checked for additional articles. A computer database for dissertations was then searched for other EI research that has not been published in journals. Finally, 11 of the prominent EI researchers were contacted via e-mail and this generated an additional 8 studies. The resulting meta-analytic database consisted of 58 studies (marked with asterisks in the References section), with sample sizes ranging from 15 to 3,086 participants. The studies consisted of samples that were predominantly conducted in the United States with female participants. For the ability-based classifications, the MEIS and its predecessors (i.e., MSCEIT refinements) were the only measures that were included. This decision was made because of the performance-based scoring format it employs. The other measures used a self-report format and were classified as mixed measures. The most common mixed measures included the Schutte et al. (1998) Emotional Intelligence Scale, Bar-On’s EQ-i, the Trait Meta Mood Scale (Salovey, Mayer, Goldman, Turvey, & Palfai, 1995), and the Emotional Competence Inventory (Sala, 2002). Although some of the studies were dissertations and technical reports, it is preferable to include all of these to be sure to provide the most accurate estimates possible. In addition, given the early stage of research on EI, many of these studies will undoubtedly end up being submitted for publication. Furthermore, meta-analyses in other domains (cf. Hunter & Schmidt,

452

VAN ROOY, VISWESVARAN, PLUTA

1990, for a summary) have not found differences between the different sources of publication at the construct-level correlations (i.e., once corrected for unreliability in the measures). Meta-Analytic Procedure The Hunter and Schmidt (1990, 2004) meta-analytic procedure was used in this study. This method provides a way to determine what portion of correlational differences across studies could be due to statistical artifacts (e.g., sampling error). The method accounts for differences in sample sizes and also provides a way to correct observed correlations for statistical artifacts. Specifically, in addition to reporting the sample size weighted mean observed correlation, we report the true score correlation that has been corrected for sampling error and unreliability in both of the measures. The reliability values used to create the distributions for each meta-analysis are reported in Table 1. Standard deviations are reported for both the uncorrected and corrected correlations. In this study, where the intent is to evaluate construct validity, the latter corrected correlations are more meaningful (Hunter & Schmidt, 1990, 2004). We also report the 90% credibility interval around the corrected mean correlation for each meta-analysis. Given that the interest is in the nomological net of EI, these values are more appropriate than confidence intervals, which are based on the uncorrected values. The interactive artifact-distribution-based meta-analysis program was used and the artifact distribution used was derived from our database. Coefficient alphas were used as estimates of reliability. TABLE 1 Reliability Averages and Ranges for the Meta-Analyses Meta-Analysis Ability-mixed Mixed-mixed Ability-GMA Ability-Openness Ability-Conscientious Ability-Extraversion Ability-Agreeableness Ability-Emotional Stability Mixed-GMA Mixed-Openness Mixed-Conscientious Mixed-Extraversion Mixed-Agreeableness Mixed-Emotional Stability

rxx M

rxx Range

ryy M

ryy Range

.90 .87 .72 .81 .83 .81 .84 .81 .86 .85 .85 .85 .86 .84

.86–.92 .82–.93 .17–.92 .60–.96 .60–.96 .60–.96 .60–.96 .60–.96 .64–.96 .64–.95 .64–.96 .64–.96 .64–.96 .64–.96

.87 .86 .78 .83 .85 .88 .80 .87 — .73 .81 .81 .76 .86

.72–.96 .78–.95 .69–.88 .70–.96 .75–.92 .83–.94 .62–.94 .80–.94 — .63– .81 .71–.86 .80–.82 .71–.83 .83–.89

Note. rxx coefficient alpha reliability for first variable listed in the model. The rxx and ryy averages in each meta-analysis were used to create the reliability artifact distribution. GMA = XXXXX.

EI CONSTRUCT VALIDITY

453

RESULTS Meta-Analysis of EI Models The overall relation between the mixed and ability models of EI was examined first to provide an estimate of convergent validity. As shown in Table 2, across 13 samples the sample size weighted mean observed correlation was .12. When corrected for unreliability in the correlated measures, the value increased to .14. However, the sample size weighted observed standard deviation was large (.16) and the 90% credibility interval included zero; both of these suggest the presence of moderators in this relation. More importantly, from our objective, the upper bound of the interval around .14 does not include 1.0 (in fact, does not even come close to 1.0), suggesting that the two models are not the same. This was followed by a meta-analysis assessing the relation between different measures of EI classified as mixed model measures. Based on 11 samples, the uncorrected and estimated true score correlations between mixed model measures of EI were .61 and .71, respectively. The difference in the true score correlations between the mixed-mixed and mixed-ability meta-analyses (i.e., .57) suggests that the two models, although sharing certain characteristics, diverge more than converge, indicating that two different constructs are being tapped. These results also lend support for the remainder of our analyses, where we group measures into ability and mixed classifications to examine the differential relation of each model with personality and cognitive ability. Personality and Cognitive Ability Clear differences emerged when comparing the overall score of each model with personality and cognitive ability (GMA). These values are shown in Table 3 for the ability model and in Table 4 for the mixed model. Using the ability approach, no true score correlations with any of the Big Five factors exceeded .20; with a mixed TABLE 2 Meta-Analysis of the Correlation Between Emotional Intelligence Measures Meta-analysis

K

N

R-bar

SDr

ρ

SDρ

% var SE

90% CI

Mixed-ability Mixed-mixed

13 11

2,442 3,156

.12 .61

.1643 .0987

.14 .71

.1654 .1001

19.3 14.7

–.08 to .35 .58 to .84

Note. K = Number of samples; N = total sample size of all studies meta-analyzed; R-bar = sample size weighted mean observed correlation; SDr = observed sample size weighted mean standard deviation; ρ = true score correlation, computed by correcting observed mean for unreliability in both measures; SDρ = standard deviation of true score correlation; %var SE = percentage of variance attributable to sampling error; 90% CI = 90% credibility interval computed as ρ ± 1.28 (SDρ).

454

VAN ROOY, VISWESVARAN, PLUTA

TABLE 3 Meta-Analysis of the Correlation Between Ability-Based Emotional Intelligence Measures With GMA and the Big Five Meta-Analysis

K

N

R-bar

SDr

ρ

SDρ

% var SE

90% CI

GMA Openness Conscientiousness Extraversion Agreeableness Emotional Stability

18 11 9 11 10 11

3,872 2,643 2,353 2,643 2,529 2,643

.25 .11 .05 .07 .15 .07

.1234 .0926 .0792 .0613 .0917 .0552

.34 .14 .06 .09 .18 .08

.1251 .0801 .0584 0 .0798 0

27.0 47.5 60.9 109.9 45.3 136.1

.18 to .50 .03 to .24 –.01 to .14 .09 to .09 .08 to .28 .08 to .08

Note. GMA = XXXXXXX; K = number of samples; N = total sample size of all studies meta-analyzed; R-bar = sample size weighted mean observed correlation; SDr = observed sample size weighted mean standard deviation; ρ = true score correlation, computed by correcting observed mean for unreliability in both measures; SDρ = standard deviation of true score correlation; %var SE = percentage of variance attributable to sampling error; 90% CI = 90% credibility interval computed as ρ ± 1.28(SDρ).

TABLE 4 Meta-Analysis of the Correlation Between Mixed-Based Emotional Intelligence Measures with GMA and the Big Five Meta-Analysis

K

N

R-bar

SDr

ρ

SDρ

%var SE

90% CI

GMA Openness Conscientiousness Extraversion Agreeableness Emotional Stability

28 25 26 25 24 27

8,514 6,367 6,339 6,367 6,238 6,800

.11 .25 .28 .30 .22 .34

.1285 .1302 .1340 .1799 .1385 .1745

.13 .32 .33 .36 .27 .40

.1403 .1446 .1419 .2023 .1531 .1926

19.5 20.4 19.5 10.1 18.2 10.1

–.05 to .31 .14 to .51 .15 to .52 .10 to .62 .08 to .47 .16 to .65

Note. GMA = XXXXX; K = number of samples; N = total sample size of all studies meta-analyzed; R-bar = sample size weighted mean observed correlation; SDr = observed sample size weighted mean standard deviation; ρ = true score correlation, computed by correcting observed mean for unreliability in both measures; SDρ = standard deviation of true score correlation; %var SE = percentage of variance attributable to sampling error; 90% CI = 90% credibility interval computed as ρ ± 1.28 (SD ρ).

approach, four of the five correlations were in excess of .30. Agreeableness, the one factor that was not greater than .30 for the mixed model, demonstrated the highest correlation with the ability measures. Thus, the difference between these true score correlations (.09) was the smallest of the Big Five. None of the 90% credibility values are negative for the mixed model, but all of these meta-analyses have substantial standard deviations, which could indicate the presence of moderators. For two of the ability model correlations with the Big Five (i.e., Extraversion and Emotional Stability), the percentage of variance accounted for by sampling er-

EI CONSTRUCT VALIDITY

455

ror exceeded 100%, most likely as a result of second-order sampling error (Hunter & Schmidt, 1990, 2004). This suggests that the studies in those meta-analyses may not be completely representative of the total population (cf. Salgado, Anderson, Moscoso, Bertua, & de Fruyt, 2003). It is also important to note that although the true score correlations of the mixed model with the Big Five are clearly higher than those for the ability model, the lower bound 90% credibility intervals are comparable. In fact, they are nearly identical for Extraversion (.10 vs. 09, respectively) and Agreeableness (.09 for both). Turning to cognitive ability, differences were again apparent. Based on 28 correlations with a sample size greater than 8,500, a sample size weighted mean correlation of .11 was found for the mixed model-cognitive ability relation. This value increased slightly to .13 after correcting for unreliability in both of the measures. For the ability model-cognitive ability relation, the sample size weighted mean correlation of .25 is based on 18 studies with nearly 4,000 individuals. After correction for unreliability, an estimated true score correlation of .34 was found, which is much higher than the value for the mixed model. However, both of these had substantial standard deviations indicating that moderators may be present. It is apparent that ability-based measures of EI have a heavier cognitive emphasis than the mixed measures. Thus, as with personality, differences emerged for the relation with cognitive ability.

DISCUSSION There has been a surge of interest in EI and a great deal of effort has gone into theoretically delineating the parameters of the construct. Recent studies have begun to show how measures of EI are related to each other as well as constructs such as personality and cognitive ability (e.g., Brackett & Mayer, 2003; Janovics & Christiansen, 2001). Despite this, no study has yet provided a large-scale analysis of its convergent and discriminant validity. Van Rooy and Viswesvaran (2004) made an early effort, but this study greatly expands on theirs in several ways. First, in this article, we report on a unique analysis that assesses the relation between measures based on the two differing models of EI. Second, the results reported here double the number of studies (18 vs. 9) Van Rooy and Viswesvaran included in their meta-analysis of cognitive ability in relation to ability EI; for mixed EI, this meta-analysis nearly triples what they reported (29 vs. 10). Perhaps more importantly, enough studies are now available that allowed us to conduct separate meta-analyses for each model with the Big Five. Van Rooy and Viswesvaran combined all studies in their analysis, which limited the information yield. Notably, the meta-analysis of mixed EI-personality in this study is greater than even the total number they included. Thus, although there is some overlap between this study and the study of Van Rooy and Viswesvaran, we believe that this study extends and

456

VAN ROOY, VISWESVARAN, PLUTA

expands theirs in critical areas and provides a unique contribution to the field. Furthermore, it is important to note that although our meta-analyses and the meta-analyses reported in Van Rooy and Viswesvaran included a similar number of studies (almost 60), the overlap is less than 50%, as most of the Van Rooy and Viswesvaran study included samples with criterion-related validities only. Thus, from a construct explication point of view, this study provides a large value-added contribution to the literature. EI is an interesting concept both from an intuitive stance and in the way it has proceeded from its conceptualization. Many constructs are multifaceted (e.g., personality), and may even have a different number of factors (e.g., Big Five vs. Seven), but they are usually still based on the same general framework. In contrast, empirical work on EI has largely been based on two frameworks. This is reasonable given that EI is still in what could be classified as its infancy stage. Nonetheless, it makes measurement more difficult when broad terminology is used and it becomes easier to lose sight of the fact that names and constructs are not always interchangeable. Our findings support past research and suggest that ability-based measures of EI are most strongly related to the personality dimensions of Agreeableness and Openness to Experience; the other personality dimensions all had true score correlations lower than .10. In contrast, mixed measures and personality factors were highly correlated. Interestingly, the lowest correlate of the mixed model, Agreeableness, was the highest correlate of the ability model. In line with the theoretical development of the ability model (Mayer et al., 1999), these measures had a moderately high correlation with cognitive ability. This is more striking given the smaller magnitude of the mixed measures when correlated with cognitive ability (.13). Yet another surprising finding in our meta-analyses was that the correlation between the ability model EI and Agreeableness was higher than that between the ability model EI and Openness to Experience. This finding is counterintuitive and does not agree with the conceptualization of McCrae (2000). Further, we found correlations between different measures of EI based on the mixed model to be .71. This suggests a substantial specific component in each measure (assessing EI according to the mixed model). The follow-up question that arises is whether the predictive validity of these measures stem from the specific components or from the shared variance (50%, i.e., the square of .71). It is unfortunate that such parallel analyses could not be conducted with measures of EI based on the ability model. The current state of the literature is such that there exists only one series of EI measures based on the ability model. If EI is to be conceptualized as a viable construct based on the ability models, alternate measures are needed. Science progresses by investigating constructs—not individual measures. It is also important to discuss the distinction between high correlations and the uniqueness of constructs. Verbal and numerical ability are highly correlated and they reflect a more common underlying construct (i.e., general ability), but are still

EI CONSTRUCT VALIDITY

457

distinct concepts. Similarly, it is possible that the correlation between measures of EI based on the mixed and ability model is low, but that the construct of EI should actually be defined by this small shared variance. Empirical research is now needed that assesses the incremental validity of a measure of EI based on one model (say, the ability model) over a measure of EI based on the alternate (i.e., mixed model) conceptualization. However, a correlation of .14 raises the serious possibility that we are measuring two different constructs here but labeling them the same (i.e., EI). This finding, coupled with the high mixed-mixed correlation, supports our decision of how to classify the measures (i.e., mixed vs. ability), but raises another concern. That is, it is difficult to tell if differences in the tests are attributable more to the measurement method used or to the psychological processes involved in the constructs. This is more problematic in light of findings that have demonstrated that peer reports of EI correlate higher with the target persons’ self-reported personality than with his or her self-reported EI (Alonso, Van Rooy, Viswesvaran, & Collier, 2004). As mentioned, it may be that a small shared space is what really constitutes EI and that other differences could result from the measurement method employed (i.e., performance based and self-report). Although our results call into question the tenability of classifying the mixed and ability models of EI as one and the same construct, it does not implicate either of the models as inferior to the other. Instead, both models may have utility and the relative value of each could depend on the context in which it is used. On the one hand, given the broad reach of the mixed model, it may be valuable in many selection contexts. Our results, taken in combination with other results that have shown acceptable levels of predictive validity for mixed measures (e.g., Van Rooy & Viswesvaran, 2004), suggest that these measures could be useful in certain organizational settings. For instance, its predictive validity is higher than most of the Big Five factors (e.g., Barrick & Mount, 1991; Hurtz & Donovan, 2000), and it has a low cognitive component, which could limit selection concerns related to group differences (Van Rooy, Alonso, & Viswesvaran, in press). Measures based on the ability model, on the other hand, may best be suited for development programs where the aim is to increase the performance of current employees; this could also apply to domains (e.g., counseling) outside of the employment setting. With the cognitive component of these tests, there is a greater likelihood of development than with mixed measures that tap into the more stable personality traits. Further, the ability-based measures of EI, to the extent that they reflect a part of general mental ability, are more likely to result in adverse impact when used in high-stakes testing. However, in a competitive market, organizations will not forego selection based on a valid predictor just because it causes adverse impact. Furthermore, although both mixed and ability model measures could be constructed as either self-report or performance-based measures, ability measures are predominantly performance based. To the extent organizations are concerned with fakability of self-reports, ability models will be more acceptable.

458

VAN ROOY, VISWESVARAN, PLUTA

It is recommended here that the development and refinement of the EI construct and measures continue, as it is entirely possible that EI has fallen into what has become known as the jingle fallacy (Hartley, 1967); that is, to accept the notion that two entities are the same if they are called by the same name. Our results indicate that researchers in the field of EI need to remain cognizant of the fact that all measures of EI may not necessarily be tapping the same construct, and, at some point, a renaming of the models could be required. Even so, the correlation between the two models is such that there are certain portions overlapping and our findings suggest that the two models of EI are not mutually exclusive. Future research should now examine how the individual dimensions of each model are related to find where the overlap exists; the same should be done with these individual dimensions for personality and GMA. This would allow for more fine-grained analyses to determine where the two models are sharing variance and where it is that they diverge from each other and related constructs.

REFERENCES References marked with an asterisk indicate studies included in the meta-analysis. *Allen, M. J. (2000). Investigating emotional intelligence in children: Exploring its relationship with cognitive intelligence. Unpublished master’s thesis, XXXXXX, XXXXX. Alonso, A., Van Rooy, D. L., Viswesvaran, C., & Collier, S. (2004, April). Emotional intelligence and an examination of self- and peer ratings. In D. L. Van Rooy & C. Viswesvaran (Cochairs), Emotional intelligence: An unemotional but intelligent evaluation of research. Symposium presented at the 19th annual conference of the Society for Industrial and Organizational Psychology, Chicago. *Austin, E. J., Saklofske, D. H., Huang, S. H. S., & McKenney, D. (2004). Measurement of trait emotional intelligence: Testing and cross-validating a modified version of Schutte et al.’s (1998) measure. Personality and Individual Differences, 36, 1855–1864. *Barchard, K. A. (2001). Emotional and social intelligence: Examining its place in the nomological network. Unpublished doctoral dissertation, The University of British Columbia, XXXXX. Barrick, M. R., & Mount, M. K. (1991). The big five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1–26. *Bar-On, R. (1997). Bar-On Emotional Quotient Inventory: Technical manual. Toronto, XXXX: Multihealth Systems. Bar-On, R. (2000). Emotional and social intelligence: Insights from the Emotional Quotient Inventory. In R. Bar-On & J. D. A. Parker (Eds.), The handbook of emotional intelligence (pp. 363–388). San Francisco: Jossey-Bass. *Batastini, S. D. (2001). The relationship among students emotional intelligence, creativity and leadership. Unpublished doctoral dissertation, Drexel University, XXXXX. *Boyce, D. A. (2001). The correlation of emotional intelligence, academic success, and cognitive ability in master’s level physical therapy students. Unpublished doctoral dissertation, Spalding University, XXXXX. *Brackett, M. A., & Mayer, J. D. (2003). Convergent, discriminant, and incremental validity of competing measures of emotional intelligence. Personality and Social Psychology Bulletin, 29, 1147–1158.

EI CONSTRUCT VALIDITY

459

*Brackett, M. A., Mayer, J. D., & Warner, R. M. (2004). Emotional intelligence and its relation to everyday behavior. Personality and Individual Differences, 36, 1387–1402. *Carrothers, R. M., Gregory, S. W., & Gallagher, T. J. (2000). Measuring emotional intelligence of medical school applicants. Academic Medicine, 75, 456–463. *Caruso, D. R., Mayer, J. D., & Salovey, P. (2002). Relation of an ability measure of emotional intelligence to personality. Journal of Personality Assessment, 79, 306–320. *Charbonneau, D., & Nicol, A. A. M. (2002). Emotional intelligence and prosocial behaviors in adolescents. Psychological Reports, 90, 361–370. *Ciarrochi, J., Chan, A. Y. C., & Bajgar, J. (2001). Measuring emotional intelligence in adolescents. Personality and Individual Differences, 31, 1105–1119. *Ciarrochi, J., Chan, A. Y. C., & Caputi, P. (2000). A critical evaluation of the emotional intelligence construct. Personality and Individual Differences, 28, 539–561. *Ciarrochi, J., Deane, F. P., & Anderson, S. (2002). Emotional intelligence moderates the relationship between stress and mental health. Personality and Individual Differences, 32, 197–209. *Coffey, E., Berenbaum, H, & Kerns, J. G. (2003). The dimensions of emotional intelligence, alexithymia, and mood awareness: Associations with personality and performance on an emotional stroop task. Cognition and Emotion, 17, 671–679. Costa, P. T., & McCrae, R. R. (1991). The NEO Five Factor Inventory: Manual. New York: Psychological Assessment Resources. Daus, C. S., & Ashkanasy, N. M. (2003). Will the real emotional intelligence please stand up? On deconstructing the emotional intelligence “debate.” The Industrial-Organizational Psychologist, 41, 69–72. *Davies, M., Stankov, L., & Roberts, R. D. (1998). Emotional intelligence: In search of an illusive construct. Journal of Personality and Social Psychology, 75, 989–1015. *Dawda, D, & Hart, S. D., (2000). Assessing emotional intelligence: Reliability and validity of the Bar-On Emotional Quotient Inventory (EQ-i) in university students. Personality and Individual Differences, 28, 797–812. *Day, A. L., & Carroll, S. A. (2004). Using an ability-based measure of emotional intelligence to predict individual performance, group performance, and group citizenship behaviors. Personality and Individual Differences, 36, 1441–1458. *Derksen, J., Kramer, I., & Katzko, M. (2002). Does a self-report measure for emotional intelligence assess something different than general intelligence? Personality and Individual Differences, 32, 37–48. *Douglas, C., Frink, D. D., & Ferris, G. R. (XXXX). Emotional intelligence as a moderator of the relationship between conscientiousness and performance. Unpublished manuscript, Florida State University, XXXXX. *Dulewicz, V., & Higgs, M. (1999). Can emotional intelligence be measured and developed? Leadership and Organization Development Journal, 20, 242–252. *Dulewicz, V., Higgs, M., & Slaski, M. (2003). Measuring emotional intelligence: Content, construct and criterion-related validity. Journal of Managerial Psychology, 5, 405–420. Eysenck, H. J. (1998). Intelligence: A new look. New Brunswick, XX: Transaction Publishers. *Fillion, F. (2001). The construct validation of two measures of emotional intelligence. Unpublished master’s thesis, University of Guelph, XXXXX. *Fox, S., & Spector, P. E. (2000). Relations of emotional intelligence, practical intelligence, general intelligence, and trait affectivity with interview outcomes: It’s not all just ’G’. Journal of Organizational Behavior, 21, 203–220. *Fukunishi, I., Wise, T. N., Sheridan, M., Shimai, S., Otake, K., Utsuki, N., et al. (2001). Validity and reliability of the Japanese version of the emotional intelligence scale among college students and psychiatric outpatients. Psychological Reports, 89, 625–632.

460

VAN ROOY, VISWESVARAN, PLUTA

*Fund, S, & Bar-On, R. (2004). Emotional intelligence and cognitive ability among the Israeli defense forces. Manuscript in preparation. Goleman, D. (1995). Emotional intelligence: Why it can matter more than IQ. New York: Bantam. *Graves, J. G. (1999). Emotional intelligence and cognitive ability: Predicting performance in job-simulated activities. Unpublished doctoral dissertation, California School of Professional Psychology, XXXXX. Hartley, E. L. (1967). Attitude research and the jangle fallacy. In C. W. Sherif & M. Sherif (Eds.), Attitude, ego-involvement, and change (pp. 88–104). Westport, CT: Greenwood. Hedlund, J., & Sternberg, R. J. (2000). Too many intelligences? Integrating social, emotional, and practical intelligence. In R. Bar-On & J. D. A. Parker (Eds.), The handbook of emotional intelligence (pp. 136–167). San Francisco: Jossey-Bass. *Henner, T. (1998). Comparing EQ-i and TMMS scales scores. Unpublished manuscript, XXXXXXXX, XXXXXX, Israel. Hough, L. M., & Ones, D. S. (2001). The structure, measurement, validity, and use of personality variables in industrial, work, and organizational psychology. In N. Anderson, D. S. Ones, H. K. Sinangil, & C. Viswesvaran (Eds.), Handbook of industrial, work & organizational psychology: Personnel psychology (pp. 233–277). Thousand Oaks, CA: Sage. Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternate predictors of job performance. Psychological Bulletin, 96, 72–98. Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting for error and bias in research findings (2nd ed.). Newbury Park, CA: Sage. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis (3rd ed.). Newbury Park, CA: Sage. Hurtz, G. M., & Donovan, J. J. (2000). Personality and job performance: The big five revisited. Journal of Applied Psychology, 85, 869–879. *Jae, J. H. (1997). Emotional intelligence and cognitive ability as predictors of job performance in the banking sector. Unpublished manuscript, Ateneo de Manila University, XXXXXX. *Janovics, J., & Christiansen, N. D. (2001, XXXX). Emotional intelligence in the workplace. Paper presented at the 16th annual conference of the Society for Industrial and Organizational Psychology, San Diego, CA. *Kohan, A. (2002). Emotional intelligence: An investigation of discriminant and concurrent validity. Unpublished doctoral dissertation, Lakehead University, Thunder Bay, Ontario, Canada. *Lam, L. T., & Kirby, S. L. (2002). Is emotional intelligence an advantage? An exploration of the impact of emotional and general intelligence on individual performance. The Journal of Social Psychology, 142, 133–143. *Lindley, L. D. (2001). Personality, other dispositional variables, and human adaptability. Unpublished doctoral dissertation, Iowa State University, XXXX. *Lopes, P. N., Brackett, M. A., Nezlek, J., Schutz, A., Sellin, I., & Salovey, P. (2004). Emotional intelligence and social interaction. Personality and Social Psychology Bulletin, 30, 1018–1034. *Lopes, P. N., Salovey, P., & Straus, R. (2003). Emotional intelligence, personality, and the perceived quality of social relationships. Personality and Individual Differences, 35, 641–658. *MacCann, C., Roberts, R. D., Matthews, G., & Zeidner, M. (2004). Consensus scoring and empirical option weighting of performance-based emotional intelligence (EI) tests. Personality and Individual Differences, 36, 645–662. *Mayer, J. D., Caruso, D. R., & Salovey, P. (1999). Emotional intelligence meets traditional standards for emotional intelligence. Intelligence, 27, 267–298. Mayer, J. D., Salovey, P., & Caruso, D. R. (1999). Models of emotional intelligence. In R. J. Sternberg (Ed.), Handbook of intelligence (pp. 396–420). Cambridge, England: Cambridge University Press. Mayer, J. D., Salovey, P., Caruso, D. R., & Sitarenios, G. (2001). Emotional intelligence as a standard intelligence. Emotion, 1, 232–242.

EI CONSTRUCT VALIDITY

461

McCrae, R. R. (2000). Emotional intelligence from the perspective of the five-factor model of personality. In R. Bar-On & J. D. A. Parker (Eds.), The handbook of emotional intelligence (pp. 263–276). San Francisco: Jossey-Bass. *Mehrabian, A. (2000). Beyond IQ: Broad-based measurement of individual success potential or “emotional intelligence.” Genetic, Social, and General Psychology Monographs, 126, 133–239. *Miller, C. E., Tristan, E., & Leasher, M. K. (2004, XXXX). Can emotional intelligence be useful to personnel psychology? In D. L. Van Rooy & C. Viswesvaran (Cochairs), Emotional intelligence: An unemotional but intelligent evaluation of research. Symposium presented at the 19th annual conference of the Society for Industrial and Organizational Psychology, Chicago. *Muresnsky, C. L. (2000). The relationship between emotional intelligence, personality, critical thinking ability, and organizational leadership performance at upper levels of management. Unpublished doctoral dissertation, George Mason University, XXXXX. *Newsome, S., Day, A. L., & Catano, V. M. (2000). Assessing the predictive validity of emotional intelligence. Personality and Individual Differences, 29, 1005–1016. *O’Connor, R. M., Jr., & Little, I. S. (2003). Revisiting the predictive validity of emotional intelligence: Self-report versus ability-based measures. Personality and Individual Differences, 35, 1893–1902. Ones, D. S. (1993). The construct validity evidence for integrity tests. Unpublished doctoral dissertation, University of Iowa, Iowa City. *Pallazza, R., & Bar-On, R. (1995). A study of the emotional intelligence of convicted criminals. Unpublished manuscript, XXXXXXX, XXXXXX. *Parker, J. (2001). Emotional intelligence across the lifespan: Conceptual and assessment issues. Paper presented at the National Association of School Psychologists. Toronto, Canada. *Pasanen, S. E. (2000). Emotional intelligence, conscientiousness and integrity as predictors of organizational citizenship behavior. Unpublished master’s thesis, The University of California, Los Angeles. *Pellitteri. J. (2002). The relationship between emotional intelligence and ego defense mechanisms. The Journal of Psychology, 136, 182–194. *Petrides, K. V., & Furnham, A. (2001). Trait emotional intelligence: Psychometric investigation with reference to established trait taxonomies. European Journal of Personality, 15, 425–448. Reilly, R. R., & Chao, G. T. (1982). Validity and fairness of some alternative employment selection procedures. Personnel Psychology, 35, 1–62. *Roberts, R. D., Zeidner, M., & Matthews, G. (2001). Does emotional intelligence meet traditional standards for an intelligence? Some new data and conclusions. Emotion, 1, 196–231. Ryan, A. M., Ployhart, R. E., & Friedel, L. A. (1998). Using personality testing to reduce adverse impact: A cautionary note. Journal of Applied Psychology, 83, 298–307. *Saklofske, D. H., Austin, E. J., & Minski, P. S. (2003). Factor structure and validity of a trait emotional intelligence measure. Personality and Individual Differences, 34, 707–721. Sala, F. (2002). Emotional competence inventory: Technical manual. XXXXX, XX: Hay Group. Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., & de Fruyt, F. (2003). International validity generalization of GMA and cognitive abilities: A European community meta-analysis. Personnel Psychology, 56, 573–605. Salovey, P., & Mayer, J. D. (1990). Emotional intelligence. Imagination, Cognition, and Intelligence, 9, 185–211. Salovey, P., Mayer, J. D., Goldman, S., Turvey, C., & Palfai, T. (1995). Emotional attention, clarity, and repair: Exploring emotional intelligence using the Trait Meta-Mood Scale. In J. W. Pennebaker (Ed.), Emotion, disclosure, and health (pp. 125–154). Washington, DC: American Psychological Association. Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262–274.

462

VAN ROOY, VISWESVARAN, PLUTA

*Schutte, N. S., Malouff., J. M., Hall, L. E., Haggerty, D. J., Cooper, J. T., Golden, C. J., et al. (1998). Development and validation of a measure of emotional intelligence. Personality and Individual Differences, 25, 167–177. *Sitarenios, G. (1999a, XXXX). Preliminary MSCEIT results (Tech. Rep. No. X). Toronto, XXXX: Multi-Health Systems Inc. *Sitarenios, G. (1999b, XXXX). MSCEIT results — 2nd report (Tech. Rep. No. X). Toronto, XXXX: Multi-Health Systems Inc. *Sitarenios, G. (2001, XXXX). Emotional intelligence and its relationship to cognitive intelligence: Examination of empirical results. Paper presented at the annual Canadian Psychological Association, XXXXXX. *Sjoberg, L. (2001a). Emotional intelligence: A psychometric analysis. European Psychologist, 6, 79–95. *Sjoberg, L. (2001b). Emotional intelligence measured in a highly competitive testing situation. Unpublished manuscript, Center for Economic Psychology, Stockholm School of Economics, Sweden. *Stewart, J. H. (1997). Practical intelligence: Assessing its convergent and discriminant validity with social, emotional, and academic intelligence. Unpublished doctoral dissertation, The University of Southern Mississippi, XXXX. Van Rooy, D. L., Alonso, A., & Viswesvaran, C. (2005). Group differences in emotional intelligence scores: Theoretical and practical implications. Personality and Individual Differences, 38, 689–700. Van Rooy, D. L., & Viswesvaran, C. (2004). Emotional intelligence: A meta-analytic investigation of predictive validity and nomological net. Journal of Vocational Behavior, 65, 71–95. *Woitaszewski, S. A. (2000). The contribution of emotional intelligence to the social and academic success of gifted adolescents. Unpublished doctoral dissertation, Ball State University, XXXXXX, XX. *Wolfradt, U., Felfe, J., & Koster, T. (2001–2002). Self-perceived emotional intelligence creative personality. Imagination, Cognition and Personality, 21, 293–309. *Wong, C., & Law, K. S. (2002). The effects of leader and follower emotional intelligence on performance and attitude: An exploratory study. The Leadership Quarterly, 13, 243–274.