The Validity of Two Brief Measures of Creative Ability

CREATIVITY RESEARCH JOURNAL, 22(1), 53–61, 2010 Copyright # Taylor & Francis Group, LLC ISSN: 1040-0419 print=1532-6934 online DOI: 10.1080/10400410903579577

The Validity of Two Brief Measures of Creative Ability Niek Althuizen ESSEC Business School, France

Berend Wierenga Rotterdam School of Management, Erasmus University, The Netherlands

John Rossiter University of Wollongong, Australia

Two brief, 15-minute measures of creative ability have become available quite recently. The Abbreviated Torrance Test for Adults (ATTA) objectively measures creative ability in terms of the creativity of answers over 3 tasks. The Abedi Test of Creativity (ATC) subjectively measures creative ability by means of a self-rating questionnaire. Our first study examined the predictive validity of both methods by correlating prospective marketing employees’ ATTA and ATC scores with the expert-judged creativity of a marketing campaign that they were asked to design. The objective ATTA test scores correlated substantially with creative performance, unlike the subjective ATC self-rating method that was much less predictively valid. A second study with current employees of a marketing agency demonstrated discriminant validity for the ATTA by revealing that ATTA scores and supervisor ratings of creative ability correlated highly for longer-term employees, but poorly for new recruits. Based on the results of these 2 studies, we propose an even shorter-scored version of the ATTA for use in business settings.

Creativity is an elusive, yet essential, part of many scientific endeavors and of human advancement in general (Kerr & Gagliardi, 2003). Although creativity is arguably the lifeblood of science, architecture, design, engineering, business, and many other professional areas, human understanding of the complex processes that underlie creative production remains fairly limited (Simonton, 2000; Sternberg & Dress, 2001). Without doubt, however, one of the main variables that determine creative production is creative ability. Research on creative ability gained momentum when J. P. Guilford delivered his presidential address to the American Psychological Association in 1950. Guilford proposed a psychometric approach for studying creative ability in a population broader than the eminent artists Correspondence should be sent to Niek Althuizen, ESSEC Business School, Department of Marketing, Office N112, Avenue Bernard Hirsch, B.P. 50105, 95021 Cergy-Pontoise Cedex, France. E-mail: [email protected]

or inventive scientists who had been the focus of early studies (see Sternberg & Lubart, 1999). The most popular of Guilford’s paper-and-pencil tasks for measuring creative ability (more accurately described as divergent thinking tasks) is the Alternate Uses test, in which test takers are asked to list as many uses for a common object (e.g., a tin can) as possible. This test made it possible to quantify creative ability and to compare individuals on a ‘‘standard’’ scale (Sternberg & Lubart, 1999). The implicit promise was that such tests could be effectively applied to identify creative individuals for the professions. Creative ability or divergent thinking tests, such as Guilford’s Alternate Uses test, can provide a good prediction of a person’s creative performance (Plucker & Renzulli, 1999; Runco, 2007). It seems self-evident that creative ideas—which are usually at a premium—are more likely to be generated by individuals who have high creative ability. A comprehensive review of applied creativity in industry by Kabanoff and Rossiter (1994)

54

ALTHUIZEN, WIERENGA, ROSSITER

concluded that when the idea-generation task is unstructured, which is by far the most typical situation in the professions, the largest factor contributing to creativity is the creative ability of the individual. The absence of creative ability measurement in personnel assessment procedures is, therefore, surprising (Kabanoff & Rossiter, 1994). Most often used are tests of I.Q. or general mental ability (Gregory, 1999; Hunter & Hunter, 1984). I.Q. tests measure convergent thinking ability. However, for generating responses that are creative, i.e., both novel and meaningful (Amabile, 1983), divergent thinking is also necessary (ChamorroPremuzic & Reichenbacher, 2008; Tardif & Sternberg, 1988). A likely reason for the absence of creative ability measurement in personnel assessment procedures is that the formal tests of creative potential that are used in education research require lengthy administration and often cumbersome judgmental scoring (Kabanoff & Rossiter, 1994; Kerr & Gagliardi, 2003). Ideally, a psychometric measure for identifying creative individuals in business settings should be brief and easy to administer and score, yet preserve the reliability and validity associated with more extensive measures. Two new brief tests of creative ability, each taking about 15 min to administer, have become available quite recently. Both tests are based conceptually on the renowned Torrance Tests of Creative Thinking (the TTCT; see next section). The TTCT has a strong theoretical foundation based on Guilford’s research and an excellent reputation in educational research— see the reviews, listed chronologically here, by Barron and Harrington (1981), Treffinger (1985), Cooper (1991), and Plucker and Renzulli (1999). The two brief tests are the Abbreviated Torrance Test for Adults (ATTA; Goff & Torrance, 2002), which is a shortened version of the TTCT; and the Abedi Test of Creativity (ATC; Abedi, 2000), which is a 56-item self-rating instrument rather than a test, designed also to substitute for the lengthier TTCT. This article reports on the predictive validity of the two TTCT-based, brief methods of measuring creative ability. Both the ATTA and the ATC are intended to measure the same underlying construct (i.e., creative ability as defined by the Torrance tests), but they use different methods (i.e., an objective test versus selfratings). In a recent overview of creativity measurement in research and practice, Kerr and Gagliardi (2003) explicitly called for research that focuses on the validation of creativity tests (i.e., other tests than the extensively validated TTCT) and on finding shorter, more easily administered tests. This article addresses these issues. The remainder of the article is organized as follows. We first provide background on the Torrance tests. This is followed by a description of the ATTA and

the ATC. Two studies are then reported. Study 1 examines the predictive validities of the ATTA and the ATC. Study 2 examines the discriminant validity of the ATTA.

THE TORRANCE TESTS To develop the TTCT, one of the most renowned scholars in creativity research, E. P. Torrance (1966, 2008) adapted and expanded J. P. Guilford’s original (1950, 1968) tests of divergent thinking ability and creative problem-solving skills (see Sternberg & Lubart, 1999). Through a variety of verbal and figural response tasks, such as generating unusual uses for common objects, listing consequences of hypothetical events, and drawing pictures from incomplete figures, the 1966 version of the TTCT, on which the ATTA test is based, measures four subskills that, together, according to Torrance’s theory, constitute a person’s creative ability. The first three subskills involve Guilford’s well-known facets of divergent thinking and are (ideational) fluency (lots of ideas), flexibility (many different ideas), and originality (unique ideas). The fourth subskill, contributed conceptually by Torrance, is elaboration, which is the ability to embellish an idea with details. In the 1984 version of the TTCT, Torrance eliminated flexibility (because it was highly correlated with fluency) and added two other scores of facets of creative ability, called abstractness of titles and resistance to premature closure (He´bert, Cramond, Neumeister, Millar, & Silvian, 2002; Torrance & Ball, 1984). In the 1984 version, the creative ideas were additionally scored in terms of 13 creative strengths, such as emotional expressiveness, internal visualization, and richness of imagery, as to better represent the breadth of creativity demonstrated in the individual’s output. As mentioned earlier, the Torrance tests have acquired an excellent reputation. According to Plucker and Renzulli (1999, p. 39), the TTCT is ‘‘by far the most commonly used’’ test of creative ability. The TTCT has also demonstrated good predictive validity with subsequent real-life creative achievements. In perhaps the most convincing study of predictive validity, a longitudinal study that followed up high school students after graduation, Torrance (1972) reported a canonical correlation of .51 between the individuals’ TTCT scores in high school and later creative achievements. However, the lengthy administration required for the TTCT (about 75 min per testee) and scoring requirements (several experts each spending about 20 min classifying the answers) make it impracticable for use in business settings.

TWO BRIEF MEASURES OF CREATIVE ABILITY

55

ATTA AND ATC MEASURES ATTA The ATTA uses three of the nine tasks from the TTCT to measure the fluency, flexibility (as in Torrance’s pre-1984 theory), originality, and elaboration subskills of creativity. ATTA total scores combine the scores on these subskills, the sum of which is called Creative Ability, with scores on 15 creativity indicators—the two other facets mentioned earlier (i.e., abstractness of titles and resistance to premature closure) and the 13 creative strengths—to form the individual’s creativity index, which finally is transformed into a total score called Creativity Level. In this article, the validity of the Creativity Level score (the four subskills scores plus the creativity indicators) is examined in comparison with the Creative Ability score (the four subskills scores only). The Creative Ability score represents our shortscored version of the ATTA. Next, we describe the tasks from which the scores are derived. The tasks for measuring an individual’s creative potential consist of one verbal response task and two figural response tasks. For each task, the experimenter reads aloud the instructions from the manual and allows exactly 3 min for completion. The first task is a verbal Just suppose . . . task in which the respondent is asked to imagine a hypothetical situation (e.g., . . . that you could walk on air) and list, in 3 min, as many problems as possible that might occur in that situation. The responses are scored for fluency (number of responses) and originality (number of uncommon responses, i.e., those not on the list of common responses for this task). The two other tasks, again each with a 3-min time limit, ask for figural responses in the form of drawings, prompted by the stimuli of, respectively, two adjacent incomplete figures (for an example of these figures and drawings, see Figure 1) and a set of nine triangles. The testee is also asked to give each of the drawings a title. The drawings, overall, are scored for fluency (number of meaningful drawings), flexibility (number of different categories of meaningful drawings), originality (number of drawn objects not on the list of common responses), and elaboration (number of embellishments of the basic drawings, such as color, shading, and other added details). ATTA creative ability. The fluency, flexibility, originality, and elaboration ratings of the responses to each task are summed across tasks to obtain total scores for the four subskills. These raw total scores are then converted into four normalized standard scores and are rescaled so that 11 ¼ low and 19 ¼ high. The rescaled scores for fluency, flexibility, originality, and elaboration are then summed to form the ATTA Creative Ability score (range: 44 to 76).

FIGURE 1 Example of an incomplete figures task and drawing responses. Both are meaningful responses and hence counted toward fluency. If a head and a hand are not on the list of common responses, then they will also count for originality. Note. These figures were invented to preserve the confidentiality of the ATTA.

ATTA creativity level. In addition to the subskills scores, there is a second set of creativity judgments of the answers to the three tasks, called the creativity indicators. There are 15 of these indicators (examples: richness and colorfulness of imagery, combination or synthesis of two or more figures, and abstractness of titles for the drawings). The summed score for the absence (0), moderate presence (1), or strong presence (2), of the 15 creativity indicators, with the sum ranging from zero to 30, is added to the creative ability score to form the Creativity Index (range: 44 to 106). The Creativity Index score is then rescaled (by contraction) and reported as the ATTA Creativity Level score (range and interpretation: 1 ¼ minimal to 7 ¼ substantial). The ATTA takes about 15 min to administer, including recording the testee’s personal details, reading aloud the instructions from the manual, and administering the three tasks. Scoring the tasks for subskills takes a trained rater on average 10 min per testee and the Creativity Indicators scoring can take up to a further 10 min. ATC The ATC (Abedi, 2000) is a 56-item questionnaire designed to measure the same four subskills of creative ability as the TTCT, but by self-ratings rather than by a test (the ATC is misleadingly named as a test). Sample items are given in the Appendix, Table A1. For each item, the respondent has to choose the best-fitting

56


self-description from three options. Varying numbers of items per subskill are used: fluency (17 items; scale range ¼ 17 to 51), flexibility (13 items; scale range ¼ 13 to 39), originality (16 items; scale range ¼ 16 to 48), and elaboration (10 items; scale range ¼ 10 to 30). The ATC Creative Ability score is computed as the (unweighted) average of the 56 items’ scores, which range from 1 ¼ low to 3 ¼ high. The average score is scaled back to the original answer range of 1 to 3, but to two decimal places (range of total scores ¼ 1.00 to 3.00). This 2-decimal summary scale is very fine-grained, allowing 168 different scores (i.e., 56 items 3 possible scores for each item). The ATC Creative Ability score is based only on the subskills and not the creativity indicators. Typical completion time for the ATC questionnaire is about 15 min. Scoring requires no expert judgment. Convergent validity of the ATC with the TTCT was examined for a sample of over 2,000 Spanish students (age range ¼ 13 to 20 years) in a study by Auzmendi, Villa, and Abedi (1996). Despite their favorable report, the convergence was in fact poor. The correlations between the fluency, originality, flexibility and elaboration subskills and their TTCT equivalents were positive but small (ranging from .24 for fluency to .05 for flexibility). However, our concern is with predictive validity. For this purpose, it does not matter whether the ATC converges with the TTCT, although that remains a theoretical concern.

STUDY 1: PREDICTIVE VALIDITIES OF THE ATTA AND THE ATC Method Participants. The participants in this study were 120 undergraduate and graduate marketing students at a university in The Netherlands (later called the full sample). Of the 120 participants, 68% were men and the age of the respondents ranged from 19 to 30 years (M ¼ 23). This sample was used to assess the psychometric properties of the ATTA and the ATC. For the predictive validity investigation, a random subsample of 20 students was asked to complete a task that provided the creative performance criterion. Procedure. Participants first completed the ATC questionnaire individually at home and took the ATTA test in small groups under supervision of the first author at a later time. Next, 20 randomly selected participants were invited to enter a contest run by a local company to design a marketing campaign. The contest required entrants to design a loyalty-enhancing sales promotion campaign for a well-known brand of beer in The Netherlands and they were told that their campaign

might be publicized as a nationwide TV commercial. Each contestant was asked to describe his or her campaign in a maximum of two pages. They worked in isolated cubicles in the university’s behavioral lab and were given 3 hr to complete the task. The contestants took between 40 and 170 min for the task (modal time taken ¼ 120 min, i.e., 2 hr). Covariates. Based on a review of the literature, we identified a number of variables that could be related to creative ability or the expert-judged creativity of the campaign proposals, namely gender (e.g., Baer, 1999; Eysenck, 1995), age (e.g., Simonton, 1997; Wu, Cheng, Ip, & McBride-Chang, 2005), education level (e.g., Cheung, Rudowicz, Yue, & Kwan, 2003), and length of the campaign proposal (e.g., Lilien, Van Bruggen, Rangaswamy, & Starke, 2004). Results Psychometric properties. After reading and discussing the scoring manual and consulting the test provider for clarification, two raters with experience in psychological measurement independently scored all 120 participants’ ATTA booklets. The interrater reliability coefficients (intraclass correlations: see Shrout & Fleiss, 1979) for the four subskills scores ranged from .88 to .97. The interrater reliability coefficient for the Creativity Indicators score (over all 15 indicators) was .85. Hence, the scores of the two raters were combined. As shown in Table 1, the 120 students had slightly higher average Creative Ability in comparison with the norms for the ATTA and the ATC (cf. King, McKee-Walker, & Broyles, 1996, who also reported above average TTCT scores for their student sample). As with the participants in the ATTA and ATC norming studies, the 120 students exhibited a wide range of creative ability. Although the TTCT (at least the Figural part) is said to be fair in terms of gender (Cramond, 1993; Kim, Cramond, & Bandalos 2006; Torrance, 1977), in our sample women scored significantly higher than men in terms of their average ATTA Creativity Level (Mmale ¼ 4.5 versus Mfemale ¼ 5.2, p ¼ .010) and average ATTA Creative Ability (Mmale ¼ 62.7 versus Mfemale ¼ 66.0, p ¼ .003). Kershner and Ledger (1985) also found that girls outperformed boys in terms of their average Creative Ability. The only other demographic trend was a significant positive relationship between the ATC scores and age (r ¼ .26, p ¼ .005). Thus, the older the students, the higher their self-rating scores of creative ability. Correlations and factor structures. To assess the relationship between the two methods for measuring

57

TWO BRIEF MEASURES OF CREATIVE ABILITY

TABLE 1 Descriptive Statistics for the ATTA Scores and the ATC Scores in Study 1 (N ¼ 120) Compared With the Norms From the Original Studies ATTA

ATC a

Study 1 Measure Creativity levelc Creative ability Fluency Flexibility Originality Elaboration Sample size

Normsb

Study 1

Norms

M

SD

M

SD

M

SD

M

SD

4.7 63.8 16.1 15.8 16.2 15.7

1.4 5.7 1.9 1.8 1.8 2.2

4.0 60.5 15.0 15.1 15.1 15.3

n.a. n.a. 1.9 2.1 1.9 1.9

– 2.38 2.35 2.43 2.42 2.32

0.20 0.27 0.28 0.21 0.23

– 2.26 2.11 2.19 2.34 2.40

n.a. 0.25 0.30 0.23 0.28

120

118d

175

2,000 approx.

a

Norms from Goff and Torrance’s (2002) manual for the ATTA. Norms from the study by Auzmendi et al. (1996) for the ATC. c ATTA Creativity Level includes the Creativity Indicators. d Two participants’ questionnaires could not be used because of missing data. b

TTCT-based creative ability, the correlations (Pearson’s r) were calculated first for the full sample (less the two participants who had missing data for the ATC, so the final N ¼ 118) between the Creativity Level, Creative Ability, and the subskills scores of the ATTA, and the Creative Ability and subskills scores of the ATC. The correlations between the ATTA measures and the ATC measures, shown in Table 2, were low and significant only for the subskill of originality (r ¼ .25, p ¼ .006). The main ATTA score, Creativity Level (which includes the Creativity Indicators), was not significantly correlated with any of the ATC scores. That the ATTA and the ACT seem to measure different things was confirmed by a factor analysis (principal components with oblique rotation) of the subskills scores measured by both methods. As shown in Table 3, two distinct factors emerged, one for the ATTA test’s subskills and the other for the ATC self-ratings of those subskills. Both methods’ subskills scores were internally consistent (a ¼ 0.72 for the ATTA and a ¼ 0.80 for the ATC). These results suggest that each method is measuring an underlying factor called creative ability but that an objective method (the ATTA test) produces a very

different estimate of the individual’s creative ability than a subjective method (the ATC self-ratings [cf. Hocevar 1981]). The question remains, however, as to which method, the ATTA or the ATC, has the better predictive validity.

Predictive validity. To estimate the respective predictive validities of the ATTA and the ATC, we focused on the random subsample of 20 participants who completed the ATTA and the ATC and then competed in the contest individually to design a creative marketing campaign. As with the participants in the full sample, these 20 individuals, on average, had slightly higher creative ability scores in comparison with the norms for the ATTA and the ATC, but they, too, exhibited a wide range of creative ability. The subsample included 7 women and 13 men, with an average age of 23 years. The creativity of each individual’s campaign proposal, called Campaign Creativity, was rated by five expert judges. Each judge rated all 20 campaign proposals on the two dimensions traditionally used to assess creative performance: novelty (three items: uniqueness,

TABLE 2 Correlations (Pearson’s r) Between the ATTA Test Scores and the ATC Questionnaire Scores for the Full Sample in Study 1 (N ¼ 118) Measure ATTA ATTA ATTA ATTA ATTA ATTA

creativity levela creative ability fluency flexibility originality elaboration

ATC Creative Ability

ATC Fluency

ATC Flexibility

ATC Originality

ATC Elaboration

.12 .17 .14 .04 .21 .10

.14 .17 .14 .08 .15 .14

.08 .11 .13 .01 .15 .13

.17 .19 .16 .06 .25 .10

.00 .02 .01 .08 .10 .04

Note. Correlation significantly different from zero at p < .05. a Creativity Level includes the Creativity Indicators.

58


TABLE 3 Factor Analysis of the Four Subskills Scores from the ATTA Test and the ATC Questionnaire for the Full Sample in Study 1 (N ¼ 118) Measures ATTA (coefficient alpha ¼ .72) Fluency Flexibility Originality Elaboration ATC (coefficient alpha ¼ .80) Fluency Flexibility Originality Elaboration Variance Explained

Factor 1

Factor 2

.86 .70 .69 .68

24.4%

.90 .85 .78 .61 34.9%

Note. Factor loadings