Identifying Invalid Computerized Cognitive Test Performance in Patients with Depression Boaz Y. Saffer1, Grant L. Iverson2, 1, Kevin Kjernisted1, 2, & Diane McIntosh1, 2 1 2
Copeman Healthcare Centre, Vancouver, BC, Canada University of British Columbia, Vancouver, BC, Canada
Please address correspondence to Grant Iverson, Ph.D., Department of Psychiatry, 2255 Westbrook Mall, University of British Columbia, Vancouver, B.C., V6T 2A1 Canada; Email:
[email protected]. Acknowledgement and Disclosure: This study was conducted at Copeman Neuroscience Centre. Funding for this study was provided by AstraZeneca Canada and the CNS Vital Signs company. The authors have had full control of all primary data, and did not enter into an agreement with the funding organizations that limited our ability to complete the research as planned or to publish the results.
Objective: Computerized cognitive testing is often used in clinical practice and research with patients with mood disorders. Computerized testing is often unsupervised; therefore, it can be difficult to determine whether a patient fully understood the instructions or might have invalidated a test for other reasons. The purpose of this study was to systematically examine the new Validity Indicators for the CNS Vital Signs battery (CNS-VS) in a sample of adult outpatients with depression who completed the test three times as part of a treatment study. Participants and Methods: Participants were 48 outpatients with depression who were enrolled in a treatment study [age= 43.8 years (SD=12.9), education= 14 years (SD=2.3), 50.0% women, and 76.1% Caucasian]. Patients completed the CNS-VS battery three times (at a one week interval and again after approximately 8 weeks). The new Validity Indicator cut-offs and a set of supplementary validity indicators created by the present authors were applied. Out-of-range normative scores (i.e., standard scores below 40) also were identified. Results: Using the 8 CNS-VS Validity Indicators, 19.6% were identified as having invalid scores at time 1, 13.0% at time 2, and 10.9% at time 3. Supplementary invalid scores were seen in 2.2%, 2.2%, and 0% at times 1, 2, and 3, respectively. Out-of-range domain scores occurred in 19.6%, 6.5%, and 10.9% at times 1, 2, and 3, respectively. Conclusion: The CNS-VS Validity Indicators correctly identify some invalid scores. There appear to be some false positives, however, and some scores that are likely invalid are missed by these indicators (i.e., false negatives). Additional research is needed to refine the interpretation of valid versus invalid CNS-VS scores.
Presented at the 41st Annual Meeting of the International Neuropsychological Society, Waikoloa, Hawaii, February 8, 2012.
Table 1. Base Rates of Out-of-Range CNS-VS Standard Scores Domains Neurocognitive Index Memory Psychomotor Speed Reaction Time Complex Attention Cognitive Flexibility Processing Speed Executive Functioning Verbal Memory Visual Memory Subtests Verbal Memory Initial Correct Hits Verbal Memory Initial Correct Passes Verbal Memory Delayed Correct Hits Verbal Memory Delayed Correct Passes Visual Memory Initial Correct Hits Visual Memory Initial Correct Passes Visual Memory Delayed Correct Hits Visual Memory Delayed Correct Passes Finger Tapping Test – Right Hand - Number of Average Taps Finger Tapping Test – Left Hand - Number of Average Taps Symbol Digit Coding Correct Symbol Digit Coding Errors Stroop Simple Reaction Time Stroop Complex Reaction Time Correct Stroop Stroop Reaction Time Correct Stroop Stroop Commission Errors Shifting Attention Test Correct Shifting Attention Test Errors Shifting Attention Test Average Correct RT Continuous Performance Task Correct Responses Continuous Performance Task Omission Errors Continuous Performance Task Commission Errors Continuous Performance Task Choice Reaction Time Correct Subjects showing >= 1 abnormal domain standard scores Subjects showing >= 2 abnormal domain standard scores Subjects showing >= 3 abnormal domain standard scores Subjects showing >= 1 abnormal subtest standard scores Subjects showing >= 2 abnormal subtest standard scores Subjects showing >= 3 abnormal subtest standard scores
Pre-Treatment
One Week
Post-Treatment
%
%
%
4.3 2.2 4.3 0.0 15.2 13.0 0.0 13.0 0.0 0.0
2.2 0.0 0.0 0.0 6.5 0.0 0.0 0.0 2.2 0.0
4.4 2.2 0.0 0.0 8.9 2.2 0.0 0.0 4.4 0.0
2.2 0.0 0.0 2.2 2.2 0.0 2.2 0.0 2.2 2.2 0.0 0.0 0.0 0.0 0.0 6.5 4.3 15.2 0.0 2.2 2.2 6.5 2.2
2.2 0.0 4.3 0.0 2.2 0.0 0.0 0.0 2.2 0.0 0.0 0.0 2.2 0.0 0.0 0.0 0.0 4.3 0.0 4.3 4.3 6.5 2.2
0.0 2.2 0.0 0.0 4.3 0.0 2.2 0.0 2.2 2.2 0.0 2.2 6.5 0.0 0.0 6.5 0.0 0.0 0.0 10.9 10.9 13.0 2.2
19.6 15.2 10.9 34.8 13.0 4.3
6.5 2.2 2.2 21.7 8.7 2.2
10.9 4.3 2.2 26.1 13.0 10.9
An out-of-range score is defined as a standard score greater than 160 or below 40. A substantial minority of patients with depression had one or more out-of-range scores the first time they took the test (19.6% for domain scores and 34.8% for subtest scores). All out-of-range scores were below 40. Fewer out-of-range scores were attained at one-week retest, most likely due to prior exposure to the test increasing familiarity and contributing to practice effects. Clinicians and researchers need to carefully examine out-of-range scores and try to determine if they are invalid and should be disregarded, or, for research purposes, whether they should be truncated (e.g., to a standard score of 40, representing 4 SDs below the mean). Extreme out-of-range scores (e.g., scores below zero) should not be left in studies; they represent extreme outliers and they will skew the results.
Page 2 of 4
Table 2. Base Rates of Invalid Scores
Current CNS-VS validity indicators included on company website Verbal Memory raw score < 30. Visual Memory raw score < 30. SDC: less than 20 correct responses. SAT errors > SAT correct responses. FTT: total taps < 40 Stroop Simple RT > Stroop Complex RT > Stroop Stroop RT Valid Stroop CPT Errors > CPT Correct Previous CNS-VS Validity indicators not included on company website Memory raw score less than 60 Stroop Simple Reaction Time > Stroop Complex Reaction Time Stroop Complex Reaction Time > Stroop Stroop Reaction Time CPT: fewer than 30 correct responses
PreTreatment
One Week
PostTreatment
%
%
%
0.0 0.0 0.0 13.0 2.2 2.2 2.2 2.2
0.0 2.2 0.0 4.3 0.0 4.3 4.3 2.2
0.0 0.0 2.2 0.0 2.2 6.7 6.7 2.2
0.0 0.0 15.2 0.0
0.0 4.3 4.3 0.0
0.0 4.4 15.6 4.4
PreTreatment
One Week
PostTreatment
%
%
%
19.6 2.2 0.0
13.0 4.3 0.0
10.9 6.5 2.2
15.2 0.0
8.7 0.0
21.7 2.2
Table 3. Summary of Base Rates of Invalid Scores
Primary CNS-VS Validity Problems (on company website) Subjects showing >= 1 primary validity problem Subjects showing >= 2 primary validity problems Subjects showing >= 3 primary validity problems Previous CNS-VS Validity Problems Subjects showing >= 1 Previous CNS-VS Validity Problems Subjects showing >= 2 Previous CNS-VS Validity Problems
The current CNS-VS Validity Indicators identified one or more invalid scores in 11-20% of the sample across the three time periods. The Shifting Attention Test had the highest rate of invalid scores. Additional research is needed to refine the interpretation of invalid scores on CNS-VS.
Page 3 of 4
Table 4. Supplementary Validity Indicators PreTreatment
One Week
PostTreatment
%
%
%
0.0 0.0 0.0 0.0 0.0 0.0
0.0 2.2 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0
0.0
0.0
2.2 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.0
0.0 0.0
0.0 0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Verbal Memory Verbal Memory Correct Hits-Immediate = 0 AND Correct Passes Immediate = 15 AND Verbal Memory Correct Initial Reaction Time = 0 Verbal Memory Correct Hits-Delay = 0 AND Correct Passes Delay = 15 AND Verbal Memory Correct Delayed Reaction Time = 0 Verbal Memory Correct Hits-Immediate = 15 AND Correct Passes Immediate = 0 Verbal Memory Correct Hits-Delay = 15 AND Correct Passes Delay = 0 Verbal Memory Correct Hits-Immediate = 0 AND Correct Passes Immediate = 0 Verbal Memory Correct Hits-Delay = 0 AND Correct Passes Delay = 0
Visual Memory Visual Memory Correct Hits-Immediate = 0 AND Correct Passes Immediate = 15 AND Visual Memory Correct Initial Reaction Time = 0 Visual Memory Correct Hits-Delay = 0 AND Correct Passes Delay = 15 AND Visual Memory Correct Delayed Reaction Time = 0 Visual Memory Correct Hits-Immediate = 15 AND Correct Passes Immediate = 0 Visual Memory Correct Hits-Delay = 15 AND Correct Passes Delay = 0 Visual Memory Correct Hits-Immediate = 0 AND Correct Passes Immediate = 0 Visual Memory Correct Hits-Delay = 0 AND Correct Passes Delay = 0
Finger Tapping Test Finger Tapping Test (FTT) Right Taps Average = 0 Finger Tapping Test (FTT) Left Taps Average = 0
Symbol Digit Coding Symbol Digit Coding (SDC) Correct Responses = 0 AND Symbol Digit Coding (SDC) Errors = 0
Stroop Test Stroop Simple Reaction Time = 0 Stroop Complex Correct = 0 AND Stroop Complex Commission Errors = 0 AND Stroop Complex Reaction Time Correct = 0 Stroop Stroop Correct = 0 AND Stroop Stroop Commission Errors = 0 AND Stroop Stroop Reaction Time Correct = 0 Stroop Complex Correct = 12 AND Stroop Complex Commission Errors = 12 Stroop Stroop Correct = 24 AND Stroop Stroop Commission Errors = 24 Stroop Complex Correct = 0 AND Stroop Complex Commission Errors = 12 AND Stroop Complex Reaction Time Correct = 0 Stroop Stroop Correct = 0 AND Stroop Stroop Commission Errors = 24 AND Stroop Stroop Reaction Time Correct = 0
Shifting Attention Test Shifting Attention Test (SAT) Correct = 0 AND Shifting Attention Test (SAT) Errors = 0 AND Shifting Attention Test (SAT) Correct Reaction Time = 0 Shifting Attention Test (SAT) Correct = 0 AND Shifting Attention Test (SAT) Errors > 0 AND Shifting Attention Test (SAT) Correct Reaction Time = 0
Continuous Performance Test Continuous Performance Test (CPT) Correct Responses = 0 AND Continuous Performance Test (CPT) Omission Errors = 40 AND Continuous Performance Test (CPT) Commission Errors = 0 AND Continuous Performance Test (CPT) Reaction Time Correct = 0 Continuous Performance Test (CPT) Correct Responses = 0 AND Continuous Performance Test (CPT) Omission Errors = 40 AND Continuous Performance Test (CPT) Commission Errors = 160 Continuous Performance Test (CPT) Correct Responses = 40 AND Continuous Performance Test (CPT) Omission Errors = 0 AND Continuous Performance Test (CPT) Commission Errors = 160
Note: We created a large number of new validity indicators, for this study, that illustrate performance that should be considered invalid because the subject did not respond properly to the test. As seen in Table 4, these validity indicators were rarely flagged in this sample.
Table 5. Summary of Supplementary Validity Indicators PreTreatment 2.2
Supplementary Validity Problems Subjects showing >= 1 Supplementary CNS-VS Validity Problems
Page 4 of 4
One Week 2.2
PostTreatment 0.0
