Computer-based testing of reflective thinking

Symbiosis of Human and Artifact Y. Anzai, K. Ogawa and H. Mori (Editors) © 1995 Elsevier Science B.V. All rights reserved.

Computer-based testing of reflective thinking: performance in 9 to 12 year old children

437

Executive control of erroneous

Uri Shafrir Department of Applied Psychology, Ontario Institute for Studies in Education, 252 Bloor St., Toronto, Ont. M5S 1V6, Canada Attention to errors was operationalized recently by the measure of post-failure reflectivity (Shafrir & Pascual-Leone, 1990). Post-failure reflective children spend long periods of time following the production of incorrect response, compared to the time they spend following the production of correct response. Shafrir and Pascual-I.eone (1990) conceptualized post-failure reflectivity as an exploratory mental executive, a spontaneously activated "debugging" procedure that helps children reexamine and correct faulty internal plans. Researchers reported that children who show high executive control following the production of erroneous performance (post-failure reflective children) scored higher than post-failure impulsive children on an inference task, as well as on other measures of intellectual functioning and academic achievement (Shafrir & Pascual-Leone, 1990); that post-failure reflectivity generalized across tasks and across domains, and that post-failure reflective children were intentional learners (Shafrir, Ogilvie & Bryson, 1990, Experiment 2); and that those children used deep planning in constructing their action plans. In other studies, children with a reading disability were found to be significantly less post-failure reflective than normal controls (Shafrir, Ogilvie & Bryson, 1990, Experiment 1; Shafrir, Siegel & Chee, 1990). Shafrir, Ogilvie and Bryson (1990) claimed that attention to errors plays an important role in learning. This study investigated additional aspects of erroneous performance; in particular, we hypothesized that executive control of performance prior to the production of erroneous response, may also play an important role in learning. A spontaneously activated "rechecking" procedure, cued by a feeling of uncertainty about the planned, impending response, may help the child to reexamine internal plans prior to the production of an incorrect response. Attention to expected error production was operationalized by pre-failure reflectivity, defined as the ratio between mean pre-failure response latency and mean response latency; pre-failure reflective children spend long periods of time prior to the production of incorrect response, compared to the time they spend prior to the production of correct response. Finally, we operationalized overall executive control of erroneous performance as behavior that is both pre-failure and postfailure reflective. The development of executive control structures for the regulation of erroneous performance may be related to the transition from concrete to formal operations (Case, 1985; Pascual-Leone, 1987; Piaget, 1950). Paying close attention to potential as well as to actual disconfirmation of an hypothesis, i.e., pre- and post-failure reflective behavior, may be a necessary part of interpropositional, hypothetico-deductive thought, of vectorial operations, and of the manipulation of abstract sets. The development of such executive control structures may also signal the emergent increase in representational competence that enables children to expand

438 the scope of task representation to include a representation of their interaction with the task in real time. The two specific hypotheses tested in this study were: (1) twelve year olds have higher overall executive control of erroneous performance, and score higher on an inference task as well as on other intellectual and academic tasks, than younger children; (2) within each of the four age groups (9, 10, 11, and 12 years old), children who have high overall executive control of their erroneous performance, score higher than children with low overall executive control of their erroneous performance on an inference task as well as on other intellectual and academic tasks.

The PAR computer-based inference task The PAR (Pattern Recognition) task was administered individually. PAR is a computerbased induction task, with 80 stimuli of repeated designs shown through bars of different colors, heights, colors + heights, and colors + heights + sounds of varying pitch, where inter-trial intervals are subject controlled. Subjects were asked to decide whether the stimulus was a repeating design; if the subject's answer was "no", he/she was asked to point to the location of the "mistake" in the design with a blinking light on the computer screen; response was immediately followed by a yes/no feedback; the subject had to strike a key in order to see the next stimulus. The unobstrusive nature of the computer-based PAR task made it possible to record both response- and post-response latencies. The response latency was decomposed into pre-success and pre-failure; similarly, the post response latency was decomposed into post-success and postfailure. Pre-failure and post-failure reflectivities were calculated from the formulae: Mean pre-failure latency Pre-failure reflectivity

(1) Mean response latency Mean post-failure latency

(2)

Post-failure reflectivity Mean post-response latency

Subjects and procedures Subjects were students in grades four through seven in five public schools in Arad, a town in the south of Israel. We tested an unselected sample of 377 subjects, aged 9 (.q = 109), 10 (.n. = 114), 11 n(.n_= 85), and 12 (.n = 69). Children in each age group were divided by a double median split on pre-failure reflectivity and on post-failure reflectivity, into 4 quadrants: children who were both pre- and post-failure reflective, were defined as having high overall executive control of erroneous performance; children who were both pre- and post-failure impulsive, were defined as having low overall executive control of erroneous performance; finally, there were two groups of mixed conditions. Scores for the Israeli version of an IQ test (M~ = 106.1, SD = 11.5), and for Raven's SPM ~ = 32.5, SD = 8.6), were obtained when children entered 3rd grade. Scores for computer-based drill and practice in arithmetic and for teachers' evaluations of the student's level of intellectual functioning (not level of academic achievement) in percentiles were available at the time of this study. The Figural Intersection Test (FIT) for attentional capacity (Pascual-Leone & Ijaz, 1989), was group-administered.

439 Results Descriptive statistics are shown in Tables 1 and 2. Results of 2-way ANOVAs, age (4 levels: 9, 10, 11, and 12 years old) by overall executive control of erroneous performance (4 levels: high, low, and two mixed conditions) are shown in Table 3.

Table 1 Mean (SD) score on PAR, pre-failure reflectivity and post-failure reflectivity by age group Age group

Measure

9 years (n = 109)

10 years (n = 114)

11 years (n = 85)

12 years (n = 69)

PAR

.59 (.18)

.60 (.20)

.63 (.20)

.76 (.12)

Pre-failure reflectivity

1.16 (.22)

1.21 (.92)

1.15 (.21)

1.34 (.42)

Post-failure reflectivity

1.60 (.55)

1.73 (.80)

1.67 (.61)

2.14 (.92)

Note. PAR = proportion of correct response Twelve year olds scored significantly higher on the FIT task for attentional capacity, showed higher overall control of erroneous performance (were more pre-failure and more postfailure reflective) than children in the 9 to 11 years age range; the 12 year olds also scored higher on PAR, and on a variety of intellectual and achievement measures. Within each age group, children with high overall executive control of erroneous performance on the PAR task, scored significantly higher on a variety of tasks of intellectual functioning, on teacher's evaluations of intellectual functioning, and on arithmetic drill and practice, than children with low overall executive control of erroneous performance; the two groups of mixed conditions scored in the intermediate range. The younger children in the 9 and 10 years old age groups with high overall executive control of erroneous performance, scored as high as the 12 year olds on a measure attentional capacity (a score of 5 on the FIT task); these younger children scored significantly higher than the children in the 12 years old age group who had low overall executive control of erroneous performance, on the various tasks. Table 4 shows a commonality analysis of the variance of the score on PAR as the dependent variable, and age, FIT, IQ, SPM, Math, and pre- and post-failure reflectivities as the independent variables. Each R 2, the squared zero-order correlations between the dependent variable and each independent variable, was partiaUed into two components: unique variance of each independent variable, and common variance shared between the particular independent variable and one or more of the other independent variables (Kerlinger & Pedhazur, 1973). The unique contribution of the overall executive control of erroneous performance (pre- and post-failure reflectivities) to the variance of the score on the PAR inference task is higher (about

440 15 %), than the unique contributions of each of the other independent variables, age (2.3 %), Mcapacity (0.8 %), IQ (2.7 %), SPM (0.6 %), and arithmetic drill and practice (0.0 %). Table 2 Mean (SD) gores on tasks of intellectual functioning and academic achievement, for the low and high groups of overall executive control of erroneous performance by age

Overall executive control of erroneous performance Low

High

Age group

9

10

11

12

9

10

11

12

Number of subjects

30

39

28

20

31

39

29

21

PAR

.41 (.11)

.42 .46 (.15).21)

.63 (.09)

.72 (.13)

.73 (.12)

73 (.14)

.83 (.08)

Pre-failure reflectivity

1.0 (.07)

1.1 (.09)

1.0 (.10)

1.0 (.11)

1.3 (.19)

1.4 (.37)

1.3 (.17)

1.7 (.48)

Post-failure reflectivity

1.2 (.12)

1.2 (.14)

1.3 (.17)

1.6 (.25)

2.1 (.70)

2.2 (.96)

2.0 (.68)

2.7 (1.10)

IQ

100.4 98.6 98.0 102.6 (10.4) (10.3) (14.6) (8.1)

110.7 108.3 104.3 109.3 (11.9) (11.4) (10.6) (11.6)

SPM

26.4 (8.3)

26.5 (7.8)

28.8 (7.2)

28.4 (5.1)

35.2 (7.4)

34.5 (6.7)

31.3 (8.3)

35.8 (6.4)

FIT

3.8 (1.3)

4.1 (1.3)

4.2 (1.5)

5.1 (1.3)

4.7 (1.3)

5.1 (1.3)

5.1 (1.5)

5.2 (1.6)

Math

-7.3 (7.1)

-10.5 -12.0 -7.4 (7.6) (14.7) (13.6)

-0.9

-1.6

-8.5

-0.2

Teacher

41.6 37.1 32.2 na (25.5) (30.8) (32.0) na

(10.9) (12.4) (11.3) (14.2) 66.8 70.2 40.8 na (23.9) (21.1) (24.4) na

Note. Low = below median for age group on both pre- and post-failure reflectivity; High = above median for age group on both pre- and post-failure reflectivity; PAR = proportion of correct response; IQ = full scale score; SPM = Raven's Standard Progressive Matrices; FIT = Figural Intersection Task; Math = arithmetic drill and practice (months ahead or behind expected grade level); Teacher teacher's evaluation of intellectual functioning (percentile).

441 Table 3 F-ratios for two-way (age X overall executive control of erroneous performance) ANOVAs for the score on PAR and other test measures, for the whole population

Measure

PAR

IQ

SPM

FIT

Math

Teacher

Age

25.7***

ns

ns

4.2**

2.8*

4.9**

Exec

107.4"**

12.6"**

18.5"**

7.8***

7.6***

16.2"**

Age X Exec

ns

ns

ns

ns

ns

ns

MS~

0.018

121.8

51.5

1.9

127.5

684.9

Note. Age = age group; Exec = overall executive control of erroneous performance; Table 4 Commonality analysis of the variance of the score on PAR as a dependent measure Independent measures Age

FIT

Unique variance

.023***

Common variance

.054

R2

.077

.008*

.107

.115

IQ

SPM

Math Prefailure

Postfailure

.027***

.006* .000

.013"*

.139"**

.155

.150

.094

.108

.219

.182

.156

.094

.121

.358

Note. ***p < .0001; *'12 < .001; "12 < .05. Conclusions These results may be interpreted in terms of the neoPiagetian constructs of the growth in attentional capacity (Pascual-I.eone, 1987), the maturation of executive control structures (Case, 1985), and optimal levels of development and skill acquisition. The results lend support to the two hypotheses. The significant increase in executive control of erroneous performance at age 12 corresponds to an increase of attentional capacity from 4 to 5 units as predicted by Pascual-Leone (1987) and to the onset of the stage of formal operations (Case, 1985; Piaget, 1950). The newly acquired ability of 12 year old children to operate on operations and not only on concrete entities facilitates the development of internal procedures that "debug" and improve

442 currently operating action plans. The longer periods of time that 12 year olds spent both prior to, as well as following the production of incorrect response, compared to the time they spent

prior to as well as following the production of correct response, appear to signal the emergence of a new type of executive control structure, aimed at optimizing performance. However, we note that the 12 year olds with low overall executive control also scored 5 on M-capacity. This unexpected result may mean that an increase in M-capacity to 5 units at age 12 is only a necessary but not a sufficient condition for the development of effective executive control. This as well as other results of this study can not easily be interpreted in terms of current neoPiagefian theory. About 1/3 of the younger children in the 9 and 10 years old age groups showed a high level of executive control of erroneous performance, and scored 5 on the FIT measure of attentional capacity, a value predicted by neoPiagetian theory for 12 years old children (Case, 1985; Pascual-Leone, 1987). These younger children also scored significantly higher on various intellectual and achievement tasks than the children in the 12 years old age group who showed low overall executive control of erroneous performance. Similar results were reported in a study of precocious cognitive development at the level of formal operations; 'psychometrically bright' 5th graders were "at a more advanced cognitive developmental level" than the 'psychometrically average' 7th graders (Keating, 1975, p. 279). Our results, showing large individual differences within age groups, suggest that a significant number of children within each age group incorporated executive control of erroneous performance in their mental representation of the task. These results suggest that this emergent representational competence is an important measure of intellectual development. However, in its present form this measure lacks the necessary metric for testing the consistency of these findings with the current framework of neoPiagetian theory. References

Case, R. (1985). Intellectual Development: Birth to Adulthood. New York: Academic Press Keating, D. P. (1975). Precocious cognitive development at the level of formal operations. Child Development, 46, 276-280. Kerlinger, F. N., & Pedhazur, E. J. (1973). Multiple regression in behavioral research. New York: Holt, Rinehart and Winston. Pascual-Lexme, J. (1987). Organismic processes for neoPiagetian theories: A dialectical, causal account of cognitive development. In A. Demetriou (Ed.), The neoPiagetian theories of cognitive development: Toward an integration (pp. 25-64). Amsterdam: North-Holland. Pascual-I_eone, J., & Ijaz, H. (1989). Mental capacity testing as a form of intellectual developmental assessment. In R. Samuda, S. Kong, J. Cummings, J. Pascual-Leone & J. Lewis (F~s.), Assessment and placement of minority students: A review for educators, (pp. 141-171). Toronto: C. J. Hogrefe. Piaget, J. (1950). The Psychology of Intelligence. London: Roufledge and Kegan. Shafrir, U., Ogilvie, M., & Bryson, M. (1990). Attention to errors and learning: Across-task and across-domain analysis of the post-failure reflectivity measure. Cognitive Development, 5, 405-425. Shaffir, U. and Pascual-Leone, J. (1990). Postfailure reflectivity/impulsivity and spontaneous attention to errors. Journal of Educational Psychology, 82, 2, 378-387. Shafrir, U., Siegel, L. S., & Chee, M. (1990). Learning disability, inferential skills and postfailure reflectivity. Journal of ~ i n g Disabilities, 23, 506-517.