Individual Differences Research 2009, Vol. 7, No. 2 , pp. 105-112 © 2009 Individual Differences Association, Inc.
www.idr-journal.com ISSN: 1541-745X
Concurrent Criterion Validity and Temporal Stability of the Robert Morris Attention Scale William E. Kelly* Robert Morris University *William Kelly; Department of Social Sciences; Robert Morris University; 6001 University Blvd; Moon Township, PA 15108-1189;
[email protected] (email). ABSTRACT – This study examined the criterion validity and temporal stability of a brief selfreport measure of attention for college students, the Robert Morris Attention Scale (RMAS). Undergraduates (N = 80) completed the RMAS and other attention-related measures, both selfreport and performance-based. The RMAS was positively correlated with maintaining focus on a stimulus and subsequent short-term recall, self-regulation, and negatively related to symptoms of attention-deficit hyperactivity disorder and deep focus to the point of non-awareness of one’s environment. These findings provide some support for the criterion validity of the RMAS.
The measurement of attention, consciously focusing mental resources onto a given stimulus while ignoring other distracting stimuli (Sternberg, 1996), has primarily been assessed using performance-based measures. For example, the Star Counting Test asks participants to count objects (stars) using specific criteria assuming that they must focus attention to the task at hand to do well (de Jong & Das-Smaal, 1995). The use of performance-based measures of attention has especially held firm in neuropsychology and experimental cognitive psychology. With the renewed interest in individual differences and personality over the past two decades (Swann & Seyle, 2005), it seems odd that the self-report measurement of attention has remained largely overlooked. Self-report has long been a staple of personality research (Larsen & Buss, 2008). This lack of development of viable selfreport measures of attention has been unfortunate as such a measure might provide a more efficient alternative to complex, time consuming performance-based measures. As such, brief self-report measures would lessen the resources demanded of both researchers and research participants. It might also further our understanding of attention into the personality/trait domain. Based on previous research, it appears difficult to operationalize attention. For example, many researchers cannot agree upon appropriate performance measures to assess attention (Moosbrugger, Goldhammer, & Schweizer, 2006; Schmidt, Trueblood, Merwin, & Durham, 1994). Hence, there is no “gold standard” performance-based measure available to study it. The instrument chosen by researchers for a given study to operationally define attention appears to primarily depend upon whether or not attention is conceptualized as an executive control or perceptual operation (for a review see Moosbrugger et al., 2006). 105
Kelly / Individual Differences Research, 2009, Vol. 7, No. 2, pp. 105-112
The lack of an oft used performance-based measure of attention as an exemplar may have been one influence that has slowed the development of a self-report attention scale; with researchers busy examining which performance measure is “best,” perhaps the examination of self-reported attention has seemed less important. Another issue that may have served to delay the development of self-report attention measures is trust that the instrument actually measures the performance criterion that it has so long been assumed to measure. This would be an issue of criterion validity (Gregory, 2000). Indeed, selfreport and performance-based measures of attention do not always converge (Paulhus, Aks, & Coren, 2001). This lack of convergence poses difficulties for researchers attempting to use self-report measures as a proxy for performance-based measures of attention. The aim of this study was to partly bridge the gap between self-report and performance-based measures by examining the criterion validity of a self-report attention scale. Most self-report measures pertinent to attention have primarily assessed AttentionDeficit/Hyperactivity Disorder (ADHD) symptoms (i.e., Murphy & Barkley, 1995). Other attention-related self-report measures have examined self-discipline and goal focus (i.e., Luszczynska, Diehl, Gutierrez-Dona, Kuusinen, & Schwarzer, 2004). Crawford (1993) devised a scale that attempted to measure specific aspects of self-reportd attention, mostly as an attempt to examine if individuals can focus enough to be hypnotized. Few researchers have developed brief, psychometrically-sound measures with the intention of directly measuring general attention. Kelly (2008) proposed a scale that showed potential as a brief, general self-report measure of attention. Based on an earlier measure that was found to relate to information acquisition on a reading task (Kelly, 2001), Kelly (2008) devised five items, the Robert Morris Attention Scale (RMAS), that had good internal consistency and a unidimensional factor structure across two samples, did not appear to evoke a socially desirable response set, and correlated with other self-report measures hypothetically related to attention such as time use efficiency, self-discipline, and academic performance. While seemingly promising as a self-report individual differences measure, Kelly (2008) did not examine the relationship between the RMAS and any performance-based measure of attention. Hence, the possibility that the RMAS might fill a gap in the literature by offering a selfreport general attention scale that converged with performance-based measures remained untested. The present study aimed to examine the criterion validity of the RMAS with both other self-report attention-related measures and a small sampling of performancebased attention measures. Temporal stability of the instrument was also examined. Method Participants and Procedure After obtaining informed consent, 80 (46 female) students enrolled in undergraduate psychology courses at a small university in the Northeast United States completed the instruments described below. The average age of the sample was 20.0 (SD = 2.0). Ninetypercent of the sample identified their ethnicity as White, 9% as African American, and 1% as Hispanic. To estimate the test-retest reliability of the scale, 59 (36 female) of the participants completed the RMAS again one-week after the initial assessment.
106
Kelly / Individual Differences Research, 2009, Vol. 7, No. 2, pp. 105-112
Instruments Robert Morris Attention Scale (RMAS). The RMAS (Kelly, 2008) is a 5-item selfreport measure of the ability to maintain attention to tasks. Participants responded using a 5-point Likert scale ranging from 1 (Strongly Disagree) to 5 (Strongly Agree). Responses to each item were summed to produce a total RMAS score. Higher scores indicated more self-reported tendency to maintain attention. Kelly reported adequate construct validity through factor analyses (extraction of a single factor) in two samples and correlations with hypothetically related measures. Reliability was reported using internal consistencies, which ranged from .76 to .82. The coefficient alpha in the current sample was .81. A sample item is “It’s easy for me to pay attention and concentrate on my activities.” Differential Attentional Processes Inventory (DAPI). Crawford (1993) developed the 40-item self-report DAPI to measure three aspects of attention: focusing attention, ignoring distractions, and multi-tasking. Participants responded using a 7-point Likert scale ranging from 1 (Never) to 7 (Always). Previous research (Lyons & Crawford, 1997) found the DAPI to encompass four factors: 1) Moderately Focused Attention, an ability to sustain moderate attention to a stimuli (i.e., “Can you concentrate on reading or studying while in a noisy room?”); 2) Extreme Focused Attention, an ability to apply one’s full attentional resources at a given time (“Can you lose yourself in thought so that you are hardly aware of the passage of time?”); 3) Dual Attention Cognitive – Cognitive Scale, the ability to carry-out two cognitive tasks simultaneously (“Can you read or study easily while at the same time listen easily to a conversation?”); 4) Dual Attention Cognitive – Physical Scale, the ability to carry out a cognitive and physical task simultaneously (“Can you talk on the telephone while doing some other physical activity?”). Responses were summed to produce factor scores and a total DAPI score. Higher scores indicated more of the specific attentional style assessed by each factor. Crawford reported the test-retest reliability of the full DAPI to be .90 (4 weeks), and coefficient alphas ranged between .91 and .94. In the current sample, the coefficient alpha was .91. Validity has been supported through factor analysis (Lyons & Crawford) and relationships with hypothetically related variables For example, Crawford, Brown, and Moon (1993) found that highly hypnotizable individuals scored higher on the Extreme Focused Attention Scale than those who were less hypnotizable. Self-Regulation Scale (SRS). The SRS (Luszczynska et al., 2004) is a 7-item selfreport scale which hypothetically measures control of attention; that is, the ability to consciously maintain or change the focus of attention. Participants responded using 4point Likert scale ranging from 1 (not at all true) to 4 (completely true). Responses were summed to produce a total SRS score, with higher scores indicating greater selfregulation. Luszczynska et al. demonstrated reliability through test-retest (.77, 1 year) and coefficient alpha estimates (.76). In the current sample the coefficient alpha was. 83. Validity was supported by confirmatory factor analysis and correlations with hypothetically related measures. A sample item is “I can control my thoughts from distracting me from the task at hand.” Young ADHD Questionnaire – Self-Report – Brief (YAQ-B). The original YAQ (Young, 2004) includes 112 items measuring four dimensions related to ADHD including ADHD symptoms (45 items), emotional difficulties (40 items), delinquency (19 items),
107
Kelly / Individual Differences Research, 2009, Vol. 7, No. 2, pp. 105-112
and social functioning (8 items). However, because only attention was pertinent to the present study, a modified 10-item version of the YAQ-SR was used to measure difficulty attending to task. This scale will be referred to as the YAQ-B. The 10 items selected had the highest factor loadings reported by Young from Factor 1 (ADHD Symptomatology) of the YAQ-SR. All items reflected difficulties maintaining attention. Participants responded using an 8-point scale ranging from 1 (not at all) to 8 (most of the time). Responses were summed to produce a total YAQ-B score, with higher scores indicating more inattention. Young reported the 45-item ADHD symptom factor of the YAQ to have a coefficient alpha of .96. The scale exhibited validity by discriminating between groups diagnosed with ADHD and controls. The coefficient alpha of the YAQ-B in the current sample was. 90. A sample item is “Have you had difficulty sustaining attention?” Digit Span and Digit Symbol. The Digit Span (Digits Forward and Digits Backward) and Digit Symbol subscales of the Wechsler Adult Intelligence Scale – Revised (WAISR; Wechsler, 1981) were used to provide a performance-based attention criterion. These subscales of the WAIS-R require individuals to focus on and perform a given task. Digit Span in particular has been used often as a measure of attention (Groth-Marnat & Baker, 2003). Digits Forward requires participants to repeat numbers in the correct order after a sequence has been read to them verbally. The sequences begin with three numbers and continue to nine numbers. As per the instructions in the WAIS-R manual, if a participant misses two items in a row, the test is discontinued. Possible scores range from 0 to 14. Digits Backward requires participants to follow the same procedure as that of Digits Forward except that after hearing the sequence, participants repeat the numbers backwards (reversed) of the order given. Digits Backward begins with two numbers and continues to a sequence of eight numbers. Possible scores range from 0 to 14. Higher scores for both subscales denote more ability to focus and repeat the sequence in the specified order. The Digit Symbol subscale is a measure of rapid symbolic coding. The test is timed and participants are instructed to “work as quickly as you can.” Participants are presented with a series of numbers paired with symbols. The task is to then correctly place the correct symbol in an empty box under the number. There are 93 numbers with which participants must match symbols. Thus, scores can range from 0 to 93. The test requires participants to attend to the task so they may quickly process information. The WAIS-R subscales were altered to be administered in a group format. The researcher read items to small groups. Participants subsequently wrote their responses rather than repeating them orally. Results Test-Retest Reliability Scores for the 59 participants who repeated the RMAS after one-week yielded a correlation of .81 between the first and second testing. This test-retest coefficient (.81) indicates satisfactory temporal stability over a short time interval (Nunnally, 1978). Also, this result suggests that self-reported attention, as measured by the RMAS, is a relatively stable trait.
108
Kelly / Individual Differences Research, 2009, Vol. 7, No. 2, pp. 105-112
Criterion Validity Correlations between all scales used in the current study are presented in Table 1. As presented in the table, the RMAS had significant positive correlations with self-regulation (SRS) and the digit forward subscale of the WAIS-R. A significant negative correlation was found between the RMAS and ADHD symptoms (YAQ-B) as well as the Extreme Focused Attention scale of the DAPI. Overall, these findings support the validity of the RMAS. However, while there was support for RMAS, in actuality the scale significantly correlated with only 40% of the other attention-related scales used in this study. This finding suggested at least two possibilities: still uncertain criterion validity of the scale, or the scales used in this study measured different constructs, or facets, of attention. Table 1 Correlations Between Measures Scale 1. RMAS 2. Digit Forward 3. Digit Backward 4. Digit Symbol 5. SRS 6. YAQ-B 7. DAPI-Total 8. DAPI-Moderate 9. DAPI-Extreme 10. DAPI- Cognitive 11. DAPI- Physical
1
2
3
4
5
6
7
1.00
.26* 1.00
.01 .33** 1.00
-.15 -.04 .09 1.00
.69** .17 -.02 .01 1.00
-.69** -.14 .04 -.02 -.63** 1.00
-.02 -.04 .17 -.02 .02 .16 1.00
8
9
.12 -.01 .15 .05 .11 -.02 .77** 1.00
-.24* -.12 .14 .07 -.22* .47** .77** .35** 1.00
10
11
.10 -.01 .05 -.07 .09 -.04 .74** .70** .38** 1.00
-.18 .12 .06 -.19 -.13 .12 .55** .31** .32** .26* 1.00
Note: N =80; RMAS = Robert Morris Attention Scale, SRS = Self-Regulation Scale, YAQ-B = Young ADHD Questionnaire – Brief, DAPI = Differential Attentional Processes Inventory. *p < .05 **p < .01
Table 2 Factor Loadings of Scales Scale RMAS Digit Forward Digit Backward Digit Symbol SRS YAQ-B DAPI-Moderate DAPI-Extreme DAPI- Cognitive DAPI- Physical
Factor 1 .88
Factor 2
Factor 3
Factor 4
.80 .81 .89 .84 -.88 .88 .64 .86 -.55
Note: N =80; RMAS = Robert Morris Attention Scale, SRS = SelfRegulation Scale, YAQ-B = Young ADHD Questionnaire – Brief, DAPI = Differential Attentional Processes Inventory.
An inspection of correlations between the measures used in this study revealed that the other measures (excluding intercorrelations of the DAPI total and its factor scales) significantly correlated with no more than 30% of other instruments. Hence, if criterion validity of the RMAS was indeterminate, it was likely that all of the scales used had questionable criterion validity. Given the unlikelihood nearly a dozen measures of what
109
Kelly / Individual Differences Research, 2009, Vol. 7, No. 2, pp. 105-112
were purported to be essentially the same construct should correlate no more than 3040% with each other, the separate construct possibility was explored. To examine if the scales measured separate attention-related constructs, a principal components factor analysis with a varimax rotation was calculated using the total scores of all measures. DAPI total scores were excluded as the factors are typically used separately (Crawford, 1993; Lyons & Crawford, 1997). Using the standard criteria of eigenvalues above 1, the results of the factor analysis yielded four factors, which combined to account for 73.1% of the systematic variance. Because of the small sample size, a minimum factor loading of .50 was used to determine if scales loaded on factors (Hair, Anderson, Tatham, & Black, 1995). Factor loadings of scales are presented in Table 2. Factor 1 accounted for 26.1% of the systematic variance in responses, included the RMAS, and seemed to reflect a general self-reported ability to maintain focus, or attention. Factor 2 accounted for 21.5% of the variance and included the DAPI scales, with the exception of the dual physical-cognitive attention scale. Factor 3 accounted for 13.6% of the variance and included the digits forward and digits backward WAIS-R scales. This seemed to indicate measurement of attention through short-term recall. Factor 4 accounted for 11.9% of the variance and included physical/perceptual-related attention elements. Discussion Overall the findings of this study offered support for the temporal stability and criterion validity of the RMAS as a measure of self-reported attention. The scale was significantly correlated with several other measures of attention. Further, when examining the first factor found in this study, the RMAS appears to have strong criterion validity regarding an ability to control one’s focus and less likelihood to be inattentive. The RMAS appears to best represent the construct of general attentional focus, which it was designed to measure. Although correlations in this study suggest only modest criterion validity for the RMAS relative to performance measures, there appears to be at least some link between self-report and performance-based attention using this particular scale. Because both the DAPI measures that loaded on Factor 2 and the RMAS, YAQ-B, and SRS in Factor 1 are all self-report instruments, the split in factors does not appear to be completely the result of methodology: self-report versus performance, although this may have partly influenced the results. Based on an analysis of DAPI items, the second factor found in this study seemed to include more specific situations of self-reported attention, rather than general tendencies. This is one possibility why the DAPI scales loaded generally on their own factor. Research on other constructs has found that specific content items account for separate variance when compared to general items (Davey, 1993). The finding that the measures used in this study seemed to be assessing different elements of attention is not new. Several studies have found separate factors of attention measures that seem to include executive functioning elements and perceptual elements of attention as well as visuo-motor and visual-spanning factors (Moosbrugger, et al., 2006; Schmidt et al., 1994). However, this is one of the few such findings which included both self-report and performance-based measures. It also provides an additional element of
110
Kelly / Individual Differences Research, 2009, Vol. 7, No. 2, pp. 105-112
attention not identified in previous studies – a general self-perceived focus factor that, although it appears similar to executive functioning, is statistically separate. The finding of this general self-report factor along with the retest reliability of the RMAS suggests the possibility of an attentiveness trait. Given the lack of a significant relationship between the RMAS and social desirability (Kelly, 2008), this trait factor does not likely represent promoting oneself favorably. It seems more likely that an attentiveness trait might correspond to conscientiousness, self-efficacy, and motivational factors. Future research is needed to test these possible correlates and determine the separateness of this possible attentiveness trait from related measures. Interestingly, the RMAS was the only self-report measure in this study to correlate with any of the performance measures. This suggests a potential viability for the RMAS as a proxy for performance-based attention measures. Additional research will be required examining the RMAS in relation to other performance-based attention measures to more thoroughly test this possibility. There are limitations of the present study which should be considered before generalizing the results. First, the sample was relatively small and homogeneous. Second, the performance measures used represented only two of many performance-based measures of attention (Schmidt et al., 1994). As noted previously, additional study of the relationship between the RMAS and other attentional performance criterions would be useful. Author Note This research was supported by a Summer Research Fellowship from Robert Morris University, Summer, 2008
References Crawford, H. J. (1993). Differential Attentional Processes Inventory: Sustained and dual attentional styles inventory that discriminates between hypnotic susceptibility level. Unpublished manuscript. Crawford, H. J., Brown, A. M., & Moon, C. E. (1993). Sustained attentional and disattentional abilities: Differences between low and high hypnotizable persons. Journal of Abnormal Psychology, 102, 534-543. Davey, G. C. L. (1993). A comparison of three worry questionnaires. Behaviour Research and Therapy, 31, 51-56. de Jong, P. F., & Das-Smaal, E. A. (1995). Attention and intelligence: The validity of the Star Counting Test. Journal of Educational Psychology, 87, 80-92. Gregory, R. J. (2000). Psychological testing: History, principles, and applications (3rd ed). Boston: Allyn & Bacon. Groth-Marnat, G., & Baker, S. (2003). Digit span as a measure of everyday attention: A study of ecological validity. Perceptual and Motor Skills, 97, 1209-1218. Hair, J. F., Anderson, R. E., Tatham, R. L., & Black. W. C. (1995). Multivariate data analysis (4th ed.). Englewood Cliffs, NJ: Prentice Hall. Kelly, W. E. (2001) Personality and time on task: the role of conscientiousness in attention and information acquisition. Educational Research Quarterly, 25, 15-21.
111
Kelly / Individual Differences Research, 2009, Vol. 7, No. 2, pp. 105-112
Kelly, W. E. (2008). Psychometric properties and correlates of the Robert Morris Attention Scale. Reading Improvement, 45, 19-25. Larson, R. J., & Buss, D. M. (2008). Personality psychology: Domains of knowledge about human nature (3rd ed). Boston: McGraw Hill. Luszczynska, A., Diehl, M., Gutierrez-Dona, B., Kuusinen, P., & Schwarzer, R. (2004). Measuring one component of dispositional self-regulation: Attention control in goal pursuit. Personality and Individual Differences, 37, 555-566. Lyons, L. C., & Crawford, H. J. (1997). Sustained attentional and disattentional abilities and arousability: Factor analysis and relationships to hypnotic susceptibility. Personality and Individual Differences, 23, 1071-1084. Moosbrugger, H., Goldhamer, F., & Schweizer, K. (2006). Latent factors underlying individual differences in attention measures. European Journal of Psychological Assessment, 22, 177-188. Murphy, K., & Barkley, R. A. (1995). Preliminary normative data on DSM-IV criteria for adults. ADHD Report, 3, 6-7. Nunnally, J. C. (1978). Psychometric theory (2nd ed). New York: McGraw Hill. Paulhus, D. L., Aks, D. J., & Coren, S. (2001). Independence of performance and selfreport measures of distractibility. Journal of Social Psychology, 130, 781-787. Schmidt, M., Trueblood, W., Merwin, M., & Durham, R. L. (1994). How much do ‘attention’ tests tell us? Archives of Clinical Neuropsychology, 9, 383-394. Sternberg, R. J. (1996). Cognitive psychology. New York: Harcourt Brace. Swann, W. B. Jr., & Seyle, C. (2005). Personality psychology’s comeback and its emerging symbiosis with social psychology. Personality and Social Psychology Bulletin, 31, 155-165. Wechsler, D. (1981). Wechsler adult intelligence scale, revised manual. San Antonio, TX: Psychological Corporation. Young, S. (2004). The YAQ-S and YAQ-I: The development of self and informant questionnaires reporting on current adult ADHD symptomatology, comorbid and associated problems. Personality and Individual Differences, 36, 1211-1223.
112
