The Test of General Reasoning Ability (TOGRA) is a timed test designed for ... deductive and inductive reasoning, auditory and visual perception, language ...
709741 research-article2017
JPAXXX10.1177/0734282917709741Journal of Psychoeducational AssessmentTest Review
Test Review
Test Review
Journal of Psychoeducational Assessment 1–4 © The Author(s) 2017 Reprints and permissions: sagepub.com/journalsPermissions.nav journals.sagepub.com/home/jpa
Reynolds, C. R. (2014). Test of General Reasoning Ability (TOGRA). Lutz, FL: Psychological Assessment Resources. Reviewed by: Hector Ramos, Suzanna J. Ramos, and Joyce E. Juntune, Texas A&M University, College Station, USA DOI: 10.1177/0734282917709741
Test Description General Description The Test of General Reasoning Ability (TOGRA) is a timed test designed for individuals aged 10 to 75 years. It assesses verbal, nonverbal, and quantitative reasoning and problem-solving skills. Specifically, the items, in a multiple-choice format, cover subsidiary cognitive processes such as deductive and inductive reasoning, auditory and visual perception, language processing, spatial skills, quantitative knowledge, visual imagery, and attention. In the literature, intelligence and general reasoning ability are related, yet distinguishable constructs. Intelligence, on one hand, reflects a broader and deeper capability to reason, solve problems, plan, think abstractly, understand complex ideas, and learn quickly (Gottfredson, 1997). On the other hand, general reasoning ability constitutes an essential element in intelligence that is corroborated by a well-known survey of 661 intelligence researchers (Snyderman & Rothman, 1987). Although the TOGRA includes items that assess verbal and nonverbal reasoning, crystallized and fluid abilities, and quantitative problem solving in item formats that use both inductive and deductive processes, the number of items measuring each component is insufficient to allow for highly reliable and accurate individual measurements of each component. Instead, item responses are summed to generate the General Reasoning Index (GRI), which reflects an overall measurement of the general factors affecting reasoning and problem-solving skills. The GRI is scaled to a mean of 100, with a standard deviation of 15. This is a common and familiar metric used predominantly in the assessment of ability and aptitude in the area of achievement. The test can be administered individually or in a group setting. Apart from the traditional pencil-and-paper format, the assessment can also be administered via computer. It requires 16 min to complete and 2 to 3 min for scoring. Formal training to administer and score the measure is not required; administration is simple and the scoring is entirely objective. However, the author has expressed that test administrators should be acquainted with good testing practices and know how to respond appropriately to examinees’ questions and concerns.
Specific Description For the booklet version, the TOGRA kit includes two equivalent alternate forms—blue and green item booklets—and answer sheets for the examinee. The examinee is given either the blue form or the green form, but not both. After conducting a univariate analysis of covariance (ANCOVA), it was found that there were no significant differences in mean score levels between individuals
2
Journal of Psychoeducational Assessment 00(0)
who took either of the forms. The primary reason for two equivalent alternate forms is to enable the examiner to retest and monitor progress without being concerned with practice effects. As for the administrator, a test manual, a blue- or green-form scoring key, and a score summary are all provided. For ease of scoring, the blue and green scoring keys are transparent overlays for the TOGRA answer sheet, which contain the correct responses. Again, these are all entirely objective. Additional materials required for the administration of the test include a pencil and scratch paper for the examinee and a timer for the examiner. The computer version of the TOGRA can be administered on-screen through PARiConnect. The instructions provided prior to beginning the test are similar to the booklet version. Both have a 16-min time limit, but in the computerized version, the computer times the administration. Test items are presented in ascending order, from least to most difficult. Examinees’ scores depend entirely on how many items they answer correctly. To calculate the raw score for the booklet version, the appropriate TOGRA scoring key (blue or green) is placed on top of the answer sheet. The transparent scoring key overlays a black box around the correct responses and enables the examiner to mark correct and incorrect responses to the left of each item. Space is also provided on the bottom of the answer sheet to record the number of correct responses. To convert the raw score to an equivalent standard score or GRI, appropriate tables (organized by examinees’ ages) are located in appendices to the test manual. In addition to the GRI values, the tables in the test manuals also provide the 90% and 95% confidence intervals (CIs), percentile ranks, stanines, z scores, and normal curve equivalents (NCEs). Qualitative descriptions for each GRI value are also provided. The TOGRA Score Summary Form is designed to allow the examiner to plot the GRI profile of the examinee. As it is possible for a single examinee to complete the TOGRA on multiple occasions, the summary form also provides a reliable change page for calculating any statistical improvement or decline. According to the test manual, the reliable change page allows clinicians to record up to six comparisons. In the computerized version, all scores are produced electronically. This version also allows the examinees to generate a Score Report that details their performance on the TOGRA, relative to the normative sample.
Technical Adequacy Standardization Sample The standardization sample consisted of 3,013 participants from 39 states across the United States. Data were collected between July 2011 and November 2012. The sample was representative of the U.S. population with regard to gender, age, ethnicity, educational attainment, and region, based on information provided by the U.S. Census Bureau. The stratification along age intervals was also representative, ranging from 10 to 75 years. There were no basal or ceiling rules in place during the standardization testing. Each participant completed as many items as he or she could, within the 16-min time limit.
Reliability Internal consistency, test–retest reliability, and alternate form consistency were all used to measure the trustworthiness of the scores produced by the TOGRA. The author assessed the internal consistency reliability by examining Cronbach’s (1951) coefficient alpha values across the 23 age groups in the sample. The median alpha reliability estimate for the GRI scores across all age groups equaled .92, with a median standard error measurement of 5.4. The author also recorded coefficient alphas for the 23 age groups, all of which substantially exceeded .80. These results support the test’s reliability across all age groups.
Test Review
3
Test–retest reliability is used to estimate the stability of scores over time. The constancy of the GRI values over time was investigated using a test–retest sample of 132 participants aged 10 to 75 years, with a median age of 29.5 years. The intervals between the two test administrations ranged from 18 to 34 days, with a median test–retest interval of 24.5 days. Corrected test–retest coefficients were higher than .77, with all but one value being in the .90s. Because the TOGRA consists of alternate forms (blue and green), the alternate form reliability coefficients were also calculated. The median value was .90; all coefficient values across all age groups exceeded .85. These values strongly support the test’s alternate form reliability.
Validity The content, construct, and criterion prediction validities were all evaluated by the author to provide ample evidence that the test was capable of measuring general reasoning ability. Both qualitative and quantitative evidence for content validity were provided. The discussion of content validity focused on a review of the literature that supported TOGRA’s foundations and assumptions, such as deductive and inductive reasoning, auditory and visual perception, language processing, spatial skills, quantitative knowledge, visual imagery, and attention; together, these skills form the foundation of the primary problem-solving functions assessed by the TOGRA. In addition, the TOGRA’s content validity was determined from the item review and selection processes. A bias panel review consisting of expert reviewers with backgrounds in school psychology, clinical psychology, Native American studies, and religious studies reviewed all TOGRA items for appropriateness within the context of their respective fields of specialty and culture. Questionable items were eliminated or revised based on information provided by the panel. The bias feedback, combined with item analysis results, confirmed that any possible racial, gender, and religious biases were minimal. Several analyses were carried out to support the assessed construct validity. Construct validity is the extent to which a test measures what it purports to evaluate. The TOGRA should not be strongly related to intelligence tests because it is a test of general reasoning ability. The author reported the correlation coefficients for nine studies comparing the TOGRA with various intelligence tests such as the Wechsler Intelligence Scale for Children–Fourth Edition (WISC-IV) and Wechsler Adult Intelligence Scale–Fourth Edition (WAIS-IV), both widely used for children and adults, respectively. The TOGRA’s GRI values correlate strongly and positively with the expected WISC-IV composites, with significant correlations ranging from a low of .47 (GRI with verbal) to a high of .64 (GRI with full scale). As for the WAIS-IV, the TOGRA’s GRI values correlated moderately strongly and positively with the WAIS-IV verbal and perceptual reasoning composites, with correlations of .40 and .56, respectively. The correlations were too low to indicate that the two intelligence tests measured the same constructs as the TOGRA. However, these scores did indicate that the intelligence tests and the TOGRA measured related but distinguishable constructs. Furthermore, there was either an insignificant or no level of correlation between the TOGRA GRI values and the working memory composites for the WISC-IV and WAIS-IV, respectively. This result is not surprising, because a direct assessment of memory is absent from the TOGRA. The overall magnitude of these correlations supports the TOGRA as a measure of reasoning ability and not as an intelligence test in disguise. When correlated with another measure of general reasoning ability, the Wonderlic Personnel Test, the mean scores across the two measures were nearly identical. Criterion prediction validity estimates how accurately an individual’s future performance can be forecast based on another test score. With regard to the relationship between general reasoning scores and the level of job complexity associated with an occupation, it was expected that as the level of job complexity increased, so would the TOGRA scores of the individuals in these jobs. The O*NET provides information on job complexity for a variety of occupations. In a subsample of 1,039 employees, there was a significant positive relationship between the level of complexity and the reasoning ability required for a particular occupation.
4
Journal of Psychoeducational Assessment 00(0)
Commentary and Recommendations The TOGRA provides rapid assessment of general reasoning and problem-solving skills. The types of items and range of difficulty of the test seem appropriate for the stated age groups. The TOGRA has been developed with a fairly large, nationally representative sample. Information regarding reliability and validity provides empirical evidence of psychometric quality. The TOGRA has several strengths worthy of mention. As stated earlier, the TOGRA consists of two equivalent forms (blue and green), allowing for the test to be utilized for later retesting and progress monitoring. The design of the TOGRA, either the traditional paper-and-pencil or computer version, maximizes the number of potential examinees. This allows the examiner’s greater flexibility in the intellectual assessment process. The administration time is relatively short, and the user has the flexibility to administer the test on an individual basis or in groups. Unlike most ability tests, the TOGRA can be administered to individuals with significant fine-motor impairments, as the measure only requires the examinee to bubble in letters on an answer sheet or use a computer mouse appropriately. The TOGRA is easy to administer, score, and interpret. The test manual is straightforward and supported by explicit diagrams and examples, so errors in administration are minimized. The TOGRA is useful to examiners in settings where a speeded measure of reasoning ability and problem solving under pressure is considered imperative in the selection process. The measure can be extended to evaluate individuals suitable for leadership roles or specialized training in any organizational arena. Furthermore, the TOGRA is an ideal measure for screening students who might be eligible for participation in school programs for gifted and talented youth. Such programs require superior levels of performance in general reasoning and problem solving. In addition, the TOGRA can serve as a diagnostic tool to identify high-ability students who may be gifted but still require problem-solving skills. Thus, intervention measures can be put in place to help students develop those skills. Although there are many strengths, the TOGRA does have some limitations. The test–retest reliability is questionable, due to its small sample size of 132 individuals ranging in age from 10 to 75 years. Although there is little to no significant evidence of item bias, testing for measurement invariance would provide a more robust indication of content validity. Another limitation is that the author did not report the effect sizes of the studies conducted for reliability and validity. Instead, the author described the effect sizes as “strong” (p. 25). Despite these relatively minor limitations, the TOGRA is a strong test and meets its stated purpose. It is very user friendly for both the administrator and examinee. The test manual is comprehensive, clear, and well-organized. The test itself can be used in a wide variety of educational and business settings. We consider this test to be a superior measure of general reasoning ability. Declaration of Conflicting Interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding The author(s) received no financial support for the research, authorship, and/or publication of this article.
References Cronbach, I. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrics, 16, 297-334. Gottfredson, L. S. (1997). Mainstream science on intelligence: An editorial with 52 signatories, history, and bibliography. Intelligence, 24, 13-23. doi:10.1016/S0160-2896(97)90011-8 Snyderman, M., & Rothman, S. (1987). Survey of expert opinion on intelligence and aptitude testing. American Psychologist, 42, 137-144. doi:10.1037/0003-066X.42.2.137
