predictive value of the dynamic assessment of

PREDICTIVE VALUE OF THE DYNAMIC ASSESSMENT OF PROCESSES INVOLVED IN READING TASKS ON SCHOOL PERFORMANCE AND PROGRESS OF LEARNING DISABILITIES CHILDREN Juan José Navarro Hidalgo University of Seville Seville / Spain [email protected]

Abstract The main objective of this paper is to establish the predictive value of a dynamic assessment (DA) device on the school performance and progress achieved by a group of subjects with special learning difficulties. We also analyse to what extent the results obtained offer additional information to the one provided by static assessment tests on reading comprehension, personal-social adjustment or cognitive performance. With this aim, we used two external criteria: (a) the assessment of academic performance and progress, and (b) the subjects’ marks in the area of Language. The participating teachers had to evaluate each of the seven proposed assessment criteria using a four-level qualitative scale. In addition, they had to assess whether, for each of those criteria, the subject had made some progress or not during the experimental phase. The sample consisted of 60 subjects (experimental group) with learning disabilities (LD), on whom the DA device was implemented, and 73 subjects (control group) also having LD. The dynamics scores obtained from the application of the EDPL device (dynamic assessment of processes involved in reading tasks) indicated predictive values on school performance and progress that were significantly higher than those shown by various static evaluation approaches to reading and the evaluation of the IQ. Keywords - Dynamic assessment, reading disabilities, predictive value.

1

INTRODUCTION

Dynamic assessment (DA) research has mainly focused on the analysis of general cognitive functions and on the finding of values that could predict, comparatively better than IQ tests, the subjects’ later performance [1]; [2]. Nevertheless, this approach has often only made it possible to obtain statistical parameters of the tests’ validity and reliability, without its having a significant impact on the subjects’ improvement and on the assessment process in the school context [3]; [4]; [5]; [6]. Even if some studies have tried to reveal the predictive power of dynamic tests on academic performance (AP) in reading or mathematics, or on extended learning situations, most of these researches have used dynamic approaches that gave preference to the decontextualisation of the tasks and of the assessment process [7]; [2]. However, many of the studies aiming to define the predictive validity of dynamic tests have been criticised for the lack of methodological rigor when drawing their conclusions [8]; [2]. Likewise, the theoretical and methodological bases on which their procedures and findings are set have also been questioned. In this sense, we can argue that the measures used in most of those studies to determine the subjects’ learning potential do not take into consideration neither the school contents nor the school context, even if their proposals are mainly directed to the latter. Results and conclusions are thus presented regarding the prediction of AP, although the connection between the assessment activities and the teaching and learning activities usually developed in school is not clear. In this sense, the decontextualisation of the tools and of the assessment process in dynamic tests, linked to the fact that the activities’ contents are not always related with the specific domains of school learning, could call into question some of the conclusions drawn regarding validity [9]; [10]; [6]; [11]. In recent years a rediscovery has taken place of the application of dynamic assessment procedures to specific domains of school learning, significantly helping to reset this field’s possible contributions to the teaching and learning process [3]; [4]; [12]; [1]; [13]; [2]; [14]; [15]; [16]. In this sense, some studies have tried to determine the predictive power of dynamic applications on reading as compared to statistical tests. For example, Hamers, Penning and Guthke [17] found that the results

obtained with the dynamic application of some Learning Tests with specific reading contents correlated to a greater extent than an intelligence test or a regular Learning Test with the pupils’ achievements in that domain. Lidz, Jepsen and Miller [18] also found significant improvements in the level of prediction on reading when they dynamically used and applied certain specific subtests selected from the Cognitive Assessment System based on the PASS model [19]; the improvements were greater than those obtained with the application of a static test. On the other hand, the relevance of the close relation between test modality – i.e. the assessment methodology adopted – and teaching methodology has also been proved. This fact confirms the test’s ecological validity as one of the essential premises of any effective proposal in the educational context. A test’s ecological validity could be determined according to its contents, but also in relation to the assessment procedure used. In this sense, the more this procedure resembles the teaching and learning procedures worked in school, the greater the ecological validity of the process and, consequently, of the results. It is particularly on this regard that DA outstands from static assessment procedures. From these aspects, we can conclude that the studies intending to evaluate the predictive capacity of dynamic tests could encounter serious problems when trying to correlate learning potential measurings – obtained in collaborative interaction situations sometimes focused on cognitive and metacognitive processes and abilities – with AP measuring possibly obtained with teaching and assessing procedures that are far from being dynamic, and even with AP static measures focused on the learning product [21]; [22]; [11]. On the contrary, when the prediction is made on extended learning situations that are conceptually and methodologically related with the contents and procedures of the DA, results seem to be more consistent and promising [5]; [23]; [24]; [2]. In other words, the greater the relation between the contents of both processes – evaluation and educational intervention – the greater as well the predictive power of the dynamic test. From these previous analyses, we infer that the DA predictive value must result in two types of indexes. On the one hand, we should be able to determine the predictive value according to the test results; on the other, we should establish the predictive value of the application process of the dynamic test. Regarding the first aspect, it is given by the existing correlation between the dynamic score (DS) – be it obtained through calculating the difference between the pretest and posttest scores, or through the score directly obtained from the dynamic tests – and a specific value considered the criterion to measure the model’s validity. This last factor may be the score obtained in the posttest, the score on other tests that are equivalent to the posttest, or any of the values gathered from the application context of the test, as for example quantifiable values of the AP. Regarding the second aspect, the predictive value of the dynamic application process derives from the relation between those methodological patterns that optimise the subject’s cognitive functioning or the personal-social adjustment during the development of the DA, and a value from the test application context. This context value – for example, AP – can at the same time be a quantifiable value of the criterion test result, or maybe a value, either quantitative or qualitative, obtained from the subject’s learning process. The establishment of the predictive value of the DA process is therefore similar to the fact of determining the mechanisms through which an educational intervention process is successful, provoking significant improvements in the subjects. The present work is focused on establishing the predictive value of a DA device applied to the processes involved in reading (EDPL) in relation to AP and progress (PR) in a group of subjects with special learning disabilities (LD). In addition, we aim to analyse to what extent the results obtained offer additional information to that procured through static evaluation tests on the subjects’ reading comprehension, personal-social adjustment and cognitive performance.

2 2.1

METHOD Participants

The sample was composed by an experimental group (EG), including 60 subjects with LD, to which the EDPL device was applied, and a control group (CG), gathering 73 subjects also having LD. Thirteen educational centres (seven primary education centres and six secondary education centres), located in Seville and Cadiz (Andalusia-Spain), took part in this research. The application of the EDPL device was done during 16 weeks by 12 teachers and school counsellors who were previously trained on the theoretical and methodological bases of this proposal, as well as on its contents.

2.2

Procedure

We have used two external criteria: (a) the teachers’ evaluation regarding the pupils’ AP and PR during the application, and (b) the marks obtained in the area of Language. We elaborated a registration sheet for each of the experimental groups with the names of all the subjects in them. Once the experimental application was finished, the teachers had to evaluate, through the use of a four-level qualitative scale, each of the seven assessment criteria proposed. These criteria were formulated according to the processes involved in reading and contemplated in our research, and to the assessment criteria for the area of Language that are especially related with reading comprehension as defined on the Educational Administration Decrees. The evaluation was proposed to the subjects’ class teachers, and to the teachers-appliers of the device (the only ones knowing the assignation of the subjects to either the EG or the CG). For the subjects in the CG the class teachers’ and support teachers’ evaluation was also requested. The qualitative scale proposed for the teachers’ evaluation was as follows: (1) Low or very low level; (2) Medium-low level, (3) Medium-high level and (4) High or very high level. The registration sheet included two boxes by the numbers of the qualitative scale that referred to the subject’s progress observed by the teacher for each of the considered criteria. The teacher had to mark X on box “P” if he/she considered that the pupil’s progress was adequate in relation to the evaluated assessment criteria, or box “NP” if he/she considered that the pupil had not progressed adequately. When the experimental phase was over, the final marks in the area of Language of those pupils participating in the study were requested. The criteria contemplated for the assessment of AP and PR in the area of Language is gathered in the following table: Table I. Criteria for the assessment of academic performance and progress in reading 1. He/she uses adequate strategies in dialogue situations in the classroom: listening, respecting the opinion of others, expressing his/her points of view, etc. 2. He/she reads and understands different types of school texts adequate to his/her level, being able to identify the topic and main ideas. 3. He/she uses planning strategies in its reading activities, identifying the task to be done and establishing objectives for them. 4. He/she poses questions or expresses his/her doubts when reading. The pupil realises when he/she does not understand, draws conclusions or comments upon the reading texts, evaluating his/her own comprehension when the reading is over. 5. He/she shows a positive attitude towards reading. 6. He/she elaborates schemes or summaries in a clear and orderly manner, capturing the text’s global sense and main ideas. 7. He/she uses written language on his/her own initiative as the means to acquire new learnings and as a source of information.

2.3

Instruments

The EDPL device is structured according to seven processes involved in reading which have been considered in our basic theory [11] and which meet the findings of scientific research in this field. These processes have been grouped into three blocks: (1) Metacognitive Processes, including: (a) metaknowledge on reading and its contents, on the strategies that may be applied and on personalsocial adjustment processes, and (b) comprehension self-regulation processes; (2) Information Analysis and Integration, including: (c) processes involved in the association of graphemes and phonemes, (d) underlying psychological processes, (e) processes involved in text integration, and (f) text-previous knowledge integration processes; and finally, (3) Processes involved in Personal-Social Adjustment. The device includes 32 assessment activities structured according to the reading processes contemplated. On the other hand, the structure of each of these activities includes: (a) the process to be evaluated, (b) the description of the activity, (c) the methodology proposed for its application, (d) a mediation and assessment proposal for metacognitive processes, and (e) the assessment indicators. To facilitate the gathering and evaluation of the mediation process, the appliers were provided with Registration and Assessment Sheets for each activity. The objective was to obtain useful information on the pupils’ learning processes, on the difficulties encountered and, especially, on the methodological patterns of mediation – i.e. those interventions applied by the evaluator that would have significantly facilitated an optimum execution of the task or that would have meant the obtaining of valuable information. In addition, the teachers-appliers had to evaluate the application of the indicators gathered at the end of each activity. These indicators were developed aiming to facilitate the analysis of the activity’s process of resolution and are related with the processes to be assessed on each of the proposed tasks. Even if their evaluation had to be mainly qualitative and oriented to the establishment of solutions for the encountered difficulties, we thought it

suitable to develop an analysis system to quantify the evaluations made. This system would allow us to make the application process more operative and to have dynamic scores (DS) for each activity and global scores available. In the research made, a series of tests has been used as the criterion to assess the impact of the device on the experimental subjects. Two of these tests have been elaborated by us according to the objectives of this study. On the one hand, we have used two text-comprehension assessment tests, the ECO 1 test, elaborated to be applied in the second and third cycles of primary education, and the ECOS test, elaborated to assess comprehension in E.S.O. (compulsory secondary education). We designed both of them [11] intending to give an answer to the assessment of knowledge and use of reading strategies, as well as to global comprehension. Tests ECO 1 and ECOS are composed of 10 texts that present various structures and that belong to different fields related with the curricular areas of primary and secondary education, respectively, followed by a series of multiple-choice questions (a total of 43 on ECO 1, and 56 on ECOS). On the other hand, the assessment of personal-social adjustment was made through the Personal-Social Adjustment Scale on Reading, APSL. This scale, also elaborated by us [11], provides information on 9 dimensions and is composed by 80 items that are presented to the pupils in a structure of short sentences to which the pupils must express their agreement or disagreement. Finally, the evaluation of cognitive performance is made using a Cattell factor “G” test, Scale 2 - Form A [25]. We must underline that the analysis on the validity and reliability of the comprehension and personal-social adjustment tests, together with the experts’ validation gathered previously to the application, provided with especially positive results.

3

RESULTS

We will now present the results of the contrasts made to assess the differential predictive value of the EDPL device in relation to other measures obtained in the study. These measures are: (a) pretest and posttest scores on the different criterion tests used, (b) improvement scores (IS) obtained from the posttest – pretest difference, (c) quantified evaluations of the class teachers and support teachers on the subjects’ AP in the area of Language, (d) quantified evaluations of the subjects’ PR on each of the assessed criteria in the Registration Sheet described above, and (e) the marks obtained (LM) in the area of Language. We made various regression analyses aiming to specify the information on the predictive value of the device. On the other hand, we adopted a qualitative differentiation originally proposed by Budoff [26], which allows to analyse those subjects in the EG who experienced significant 1 improvements on the posttest starting from low scores (≤ 84) in the criterion tests (gainers), and those who, starting as well from low scores, did not experienced any gains (non-gainers). The dynamic measures contemplated in our analysis are: (1) global average score obtained on the EDPL device (DS) and (2) the improvement score (IS). Considering posttest score as a post-instruction measuring, it has been used by several researchers to measure learning potential [27]; [22]. In this sense, although we used it as a means to compare, especially in what concerns the initial score, we must not forget that it is a static measure. The two dynamic measures contemplated should contribute with additional significance to the prediction of the subjects’ AP, in relation to the information obtained with the criterion tests. The table 1 presents some descriptive data corresponding to the EG and the CG on the various evaluations made. Immediately after it, we begin the analysis of the predictive value on each of the tests. The sizes of the samples in the different contrasts correspond to the ones reflected in this first table. The regression analysis of the EG showed that the DS significantly predicted the AP evaluated by the class teachers (R² = 0.379; F = 18.323, p < 0.000). The results of the stepwise regression analysis revealed that only this score entered the equation, explaining 38% of the observed variance (OV); the rest of variables were excluded because they did not meet the probability value criterion (p < 0.05) for test F. The regression analysis of the class teachers’ evaluation of PR showed the inclusion of two 2 variables into the equation: first, the DS, with an R value of 0.188 (f = 6.464, p < 0.017) and second, 2 the pretest, producing a 0.150 increment of the R . The combination of both explained 33.8% of the PR observed in the subjects (F = 6.866, p < 0.004). Regarding the AP evaluated by the support teachers, the IS predicted 27.2% of the OV (F = 7.089, p < 0.015), while the DS predicted 19.6% of it (F = 4.883, p < 0.039) and the posttest 15.4% of it (F = 3.453, p < 0.079). When all the variables were 2 introduced in a stepwise regression analysis, only the IS met the entry criterion (R = 0.272).

1

All the scores in this research were transformed through the application of a derived scale with an average value of 100 and a standard deviation value of 16 [28].

Table 2. Values obtained by the EG and the CG in the various evaluations made. Measures observed

EG Average (ECO 1)

CG Average (ECO 1)

EG Average (ECOS)

CG Average (ECOS)

EG Average (APSL)

CG Average (APSL)

EG Average (Cattell)

CG Average (Cattell)

1.830 (n = 32)

1.667 (n = 27)

2.402 (n = 27)

2.085 (n = 27)

2.092 (n = 59)

1.876 (n = 54)

2.092 (n = 59)

1.876 (n = 54)

2.810 (n = 32)

2.330 (n = 27)

4.78 (n = 27)

2.59 (n = 27)

3.71 (n = 59)

2.46 (n = 54)

3.71 (n = 59)

2.46 (n = 54)

2.084 (n = 22)

1.735 (n = 7)

2.316 (n = 19)

---

2.192 (n = 41)

1.735 (n = 7)

2.192 (n = 41)

1.735 (n = 7)

4.95 ( n = 22)

4.00 (n = 7)

4.26 (n = 19)

---

4.63 (n = 41)

4.00 (n = 7)

4.63 (n = 41)

4.00 (n = 7)

4.42 (n = 19)

4.23 (n = 22)

5.74 (n = 27)

4,64 (n = 22)

5.20 (n = 46)

4.43 (n = 44)

5.20 (n = 46)

4.43 (n = 44)

Class teacher’s evaluation of the AP (over 4) Class teacher’s evaluation of the PR (over 7) Support teacher’s evaluation of the AP (over 4) Support teacher’s evaluation of the PR (over 7) Language marks (over 10)

3.1 Predictive value of the ECO 1 comprehension test Table 3. Correlation between the values and the scores obtained by the EG in the different measures contemplated EDPL DS EDPL DS

ECO 1 Pretest

ECO 1 Posttest

IS

Class teacher’s (AP)

Class teacher’s (PR)

Support teacher’s (AP)

Support teacher’s (PR)

Language marks

---

ECO 1 Pretest

-0,503**

---

ECO1 Posttest

-0,198

0,431*

---

-0,466**

0,597**

---

-0,201

0,120

0,320

---

0,178

-0,149

-0,266

0,641**

---

-0,127

0,392

0,521*

0,361

0,179

---

0,031

0,280

0,438*

0,180

0,161

0,591**

---

0,449

-0,060

-0,376

-0,312

0,109

-0,542*

-0,111

IS 0,228 Post – Pret. Class 0,616** teacher’s (AP) Class 0,391* teacher’s (PR) Support 0,443* teacher’s (AP) Support 0,282 teacher’s (PR) Language -0,761** marks * p < 0.05; ** p < 0.01

---

The evaluation of PR made by the support teachers was equally submitted to the stepwise regression analysis; only the DS entered the equation (R² = 0.215; F = 5.208, p < 0.034). The pretest score (r = -0.142), the posttest score (r = 0.280) and the IS, with a quite high correlation value (r = 0.438), were all excluded. On the other hand, the pretest (F = 4.281, p < 0.054) explained 20.1% of the performance evaluated by the LM, although none of the variables was introduced in the equation when the stepwise regression analysis was made. Concerning the posttest score, once the variables were individually introduced, the results of the regression analysis showed a value of R² = 0.356 (F = 15.511, p < 0.000) for the IS and a value of R² = 0.186 (F = 6.389, p < 0,017) for the pretest. The 2 stepwise regression analysis first highlighted the IS, producing a significant increment in R when the pretest was introduced. On the other hand, in relation to the set of scores that better explained IS, the results showed that the pretest negatively correlated with this score, while the posttest predicted it by 35.6% (F = 15.511; p < 0.000). The DS explained the IS with a value of R² = 0.052 that was not really significant, but stepwise regression analyses showed that the posttest and the DS jointly explained 2 48.2% of the OV, producing a 0.125 increment of the R when the DS was introduced into the equation. Regarding the CG, the teachers’ estimations of its AP and PR were below those obtained for the EG, although the contrasts of the averages made through the t test for independent samples did not reveal significant statistical differences. On the other hand, the analysis of the CG data revealed us a negative correlation value between AP and the pretest (r = -0.129), while it was positive and significant in relation to the posttest (r = 0.453). LM also showed a negative correlation with the pretest and obtained positive and significant values with the posttest. Likewise, LM significantly correlated

with the IS and with the class teachers’ estimation of AP, as well as with the PR observed (r = 0.589 and r = 0.587 respectively). The regression analysis for the groups of gainers and non-gainers For the gainers subgroup (n = 8), the stepwise regression analysis regarding the AP evaluated by the teachers did not show any variable that met the entry criterion on the equation. The individual introduction had previously shown that the DS was the variable obtaining a higher explanatory value (R² = 0.306; F = 2.646, p < 0.155). In relation to the posttests, the analyses showed that only the IS could individually explain the OV among gainers: it did so by 74.2% (F = 17.293, p < 0.006). The stepwise regression analysis showed that the combination of the IS and the pretest – just as it happened with the EG as a whole – completely explained the variance, provoking a 0.258 increment 2 of R when the pretest was introduced. On the other hand, the correlation between the DS and the 2 posttest was positive (r = 0.313), with R = 0.098. In relation to the non-gainers subgroup, we present the calculations made for the group of subjects that did not obtain any posttest gains independently of their initial score (n = 16), since only 3 subjects did not improve starting from low scores. The results of the stepwise regression analysis showed that the DS predicted 61.7% of the AP, being the only variable introduced in the equation (F = 22.588, p < 0.000). In the same way, for the AP evaluated by the support teachers (n = 10), the DS was the only variable included in the equation, explaining 44% 2 of the OV (F = 6,398, p < 0,035). For this group, the IS explained the posttest with R = 0.451 (F = 11,481, p < 0,004). On the other hand, the DS negatively correlated with the posttest, while the pretest predicted 29% of the final score (F = 5,713, p < 0,031). The stepwise regression analysis again revealed that the combination of the IS and the pretest explained the totality of the OV in the posttest. 2 The first variable introduced was the IS, producing a 0.549 increment in R when the pretest entered the equation.

3.2 Predictive value of the ECOS comprehension test Table 4. Correlation between the values and scores obtained by the EG in the various measures contemplated EDPL DS

ECOS Pretest

ECOS Posttest

IS

Class teacher’s (AP)

Class teacher’s (PR)

Support teacher’s (AP)

Support teacher’s (PR)

EDPL DS

---

ECOS Pretest

-0.220

---

ECOS Posttest

-0.159

0.791**

---

-0.048

-0.018

0.598**

---

0.350

0.308

0.302

0.097

---

0.388*

0.246

0.171

0.016

0.764**

---

0.489*

0.055

0.091

0.047

1.000**

0.741**

---

0.630**

-0.303

-0.138

-0.005

0.741**

1.000**

0.741**

---

0.242

0.253

0.123

-0.168

0.683**

0.686**

0.685**

0.659**

IS Post – Pret. Class teacher’s (AP) Class teacher’s (PR) Support teacher’s (AP) Support teacher’s (PR) Language marks

Language marks

---

* p < 0.05; ** p < 0.01

In relation to the AP evaluated by the class teachers, the analysis made in this phase through the joint introduction of the variables into the equation showed a compound model that explained 32.8% of the OV (p < 0.043). On the contrary, the stepwise regression analysis showed that only the DS met the entry criterion to the equation (R² = 0.182; F = 4,884, p < 0.038 for 23 degrees of freedom). The analyses made in relation to PR showed that only the DS entered the equation (R² = 0.26; F = 7.745, p < 0,011). We obtained a similar outcome when we made stepwise regression analyses in relation to the AP and the PR evaluated by the support teachers. In this case, the predictive values of the DS were, in relation to AP, R² = 0.291 (F = 6.142, p < 0.026) and in relation to PR, R² = 0.487 (F = 14.252, p < 0.002). In this last case, the pretest, which negatively correlated with PR, entered the equation 2 producing a 0.157 increment of R . The model jointly explained 64.4% of the PR observed in the subjects (F = 12.686, p < 0.001). On the other hand, the stepwise regression analysis made in relation to LM did not show any variable meeting the significance criterion. The model explaining the outcome

2

of the posttest included, first, the pretest (R = 0.626) and, later on, the IS – the latter causing a 0.374 2 increment of R , thus giving rise to a very significant model (F = 38.461; p < 0.000). Finally, our analyses were focused on trying to explain the IS in this phase. In this case, the outcome showed that the DS was not included in the equation, remaining just the posttest as explanatory factor (R² = 0.357; F = 12.773, p < 0.002). In relation to the CG, the analyses showed significant correlation values between the AP evaluated by the class teacher and the IS (r = 0.597**), and between AP and the posttest (r = 0.672**). On the other hand, the pretest significantly correlated with the posttest (r = 747**) and also positively with the AP (r = 0.304), although not as significantly. The contrast with the evaluation of the support teachers could not be made, since the values could not be gathered for this group. In relation to LM, these negatively correlated with the pretest (r = -0.072) and positively, although with low and moderate coefficients, with the rest of values and scores: with the posttest (r = 0.137), with the IS (r = 0.314), with the AP evaluated by the class teachers (r = 0.375) and, finally, with the evaluation of PR (r = 0.223). On the other hand, we found notable differences between the evaluations made by the teachers in the EG and in the CG regarding their AP and PR, as well as regarding their LM. As we were able to observe in Table 2, the EG received perceptibly higher average estimations than those obtained by the CG. In this sense, the class teachers’ evaluation of PR in the EG was 4.78 while it only reached 2.59 for the CG. The contrast through the t test for independent samples showed very significant differences between the averages (t = 3.134, p < 0.003 for 52 degrees of freedom). LM also established very significant differences between the two groups: while the EG obtained a score of 5.74, the CG obtained only 4.64. The t value for the contrast made, on which variances were not assumed equal, was 3.291 (p < 0.002 for 40.7 degrees of freedom). The regression analysis for the gainers and non-gainers subgroups For the gainers subgroup (n = 11), the stepwise regression analysis of the AP evaluated by the class teachers did not show any variable included in the equation. The IS appeared as the factor that better predicted AP, obtaining a correlation value of r = 0.516 (p < 0,052). The DS obtained a positive correlation with AP (r = 0.397), just like the posttest (r = 0.338). On the other hand, the pretest negatively correlated with AP (r = -0.077). In relation to the PR observed, the analyses showed that only the DS met the entry criterion (R² = 0.616; F = 14,448, p < 0,004). The rest of variables were excluded from the model, with correlation values that were negative (r = -0.156 for the pretest) or moderate (r = 0.198 for the posttest and r = 0.375 for the IS). Meanwhile, the analyses in relation to the AP evaluated by the support teachers for the gainers (n = 10) did not show the inclusion of any variable into the equation. The highest correlation, close to statistical significance, was the one obtained for the IS (r = 0.488; p < 0.076), followed by the DS (r = 0.357; p < 0.156). In relation to the evaluation of PR made by the support teachers, the stepwise regression analysis showed the inclusion of the DS as the only variable in the equation (R² = 0.619; F = 13.024, p < 0.007). The posttest positively correlated with PR (r = 0.172), just like the IS (r = 0.356), while the pretest, as it happened with AP, negatively correlated with PR (r = -0.177). Regarding LM, regression analyses did not show any variable in the equation; the pretest negatively correlated (r = -0.453), just like the posttest (r = -0.071). On the contrary, the DS (r = 0.128) and the IS (r = 0.199) obtained positive values. In relation to the posttest, regression analyses showed the inclusion of two variables into the 2 2 model: first, the IS (R = 0.801), and second, the pretest, causing a 0.199 increment of R . Both variables explained the whole OV in the posttest. The DS, on the other hand, obtained a medium correlation value with the posttest (r = 0.380) and it explained on its own 14.4% of the variance. The non-gainers subgroup consisted of a reduced sample (n = 2) that did not allow us to draw relevant conclusions neither from the evaluations made nor from the observed correlations.

3.3 Predictive value in relation to the APSL scale The results of the analysis for the CG revealed a very significant correlation between the pretest and the posttest for the APSL scale (r = 0.557**), just as between the IS and the posttest (r = 0.481**), while the pretest and the IS kept a significant but negative correlation (r = -0.461**). The pretest significantly correlated with the class teachers’ evaluation of AP (r = 0.400**) and PR (r = 0.511**), while the posttest obtained non-significant medium correlation levels, and the IS showed negative correlation values. The pretest obtained higher correlation levels with the LM (r = 0.159) than the posttest (r = 0.088). The regression analysis made for the CG in relation to its AP showed the pretest as the only variable in the equation (R² = 0.213; F = 11.128, p < 0.002). For the EG, the equation contemplated only the two DS, excluding the rest of the variables. First the global DS was introduced (R² = 0.309; F = 22.763, p < 0.000) and later on, the adjustment DS, which caused a 0.082 increment

Table 4. Correlation between the values and scores obtained by the EG in the different measures contemplated Adjustment DS

EDPL DS

APSL Pretest

APSL Posttest

IS

Class teacher’s (AP)

Class teacher’s (PR)

Support teacher’s (AP)

Support teacher’s (PR)

Adjustment DS

---

EDPL DS

0.818**

---

APSL Pretest

0.075

0.169

---

APSL Posttest

0.132

0.145

0.680**

---

0.030

-0.349**

0.450**

---

0.576**

0.187

0.228

0.137

---

0.466**

0.268*

0.192

-0.012

0.733**

---

0.479**

0.229

0.189

0.039

0.693**

0.520**

---

0.274

0.351*

0.247

-0.030

0.348*

0.471**

0.627**

---

0.254

0.108

0.333*

0.296

0.586**

0.681**

0.436**

0.364*

IS 0.058 Post – Pret. Class 0.292* teacher’s (AP) Class 0.148 teacher’s (PR) Support 0.324* teacher’s (AP) Support 0.232 teacher’s (PR) Language 0.062 marks * p < 0.05; ** p < 0.01 2

LM

---

of the R . The two variables jointly explained 39.1% of the AP evaluated by the class teachers (F = 16.048, p < 0.000). In relation to the PR observed by the class teachers, the analysis for the CG showed an equation with the pretest as the only predicting variable (R² = 0.369; F = 23.986, p < 0.000); on the other hand, the analysis for the EG showed again an equation consisting of the two DS. 2 First of all the global DS (R = 0.191), a value which raised the introduction of the adjustment DS by 0.148. Jointly the two variables explained 33.9% of the PR (F = 12.811, p < 0.000). In relation to the AP evaluated by the support teachers, the results showed that only the DS was introduced into the equation (R² = 0.224; F = 10.653; p < 0.002). On the other hand, the analysis for the CG pointed to the posttest as the better explanatory factor (R² = de 0.745; F = 14.614; p < 0.012). The stepwise regression analysis made on the PR observed by the support teachers in the EG, showed the pretest as the only variable in the equation (R² = 0.102; F = 4.220, p < 0.047). The joint introduction of the two DS (global DS and adjustment DS) explained 7.5% of the OV, but the model did not proved significant. For the CG, the analysis made did not show any variable able to meet the entry criterion. Regarding LM, the stepwise regression analysis for the EG initially revealed the posttest as the only factor integrated into the equation, explaining 11.1% of the OV (F = 5.260, p < 0.027). However, when we introduced the dynamic variables together (global DS + adjustment DS), the resulting model explained 14.6% of the LM (F = 3.669, p < 0.034). The model consisting of the global DS and the IS turned out to be also significant (R² = 0.141; F = 3.353, p < 0.045). On the other hand, the analysis made for the CG did not show any significant factor capable of explaining the LM. In the EG, the posttest score was explained by the combination of the pretest and the IS. The equation introduced the pretest first (R² = 2 0.462; F = 44.655, p < 0.000) and the IS second, increasing R by 0.538. On the other hand, both DS explained only 2.2% of the final outcome, but together with the IS they could explain up to 22.7% of the posttest, the model thus becoming significant (F = 4.904; p < 0.005). For the CG the analysis 2 made produced similar results, the pretest initially explaining 31% of the posttest and incrementing R by 0.69 with the IS. Finally, in the regression analysis made for the EG regarding the attitude evaluated by the class teachers, it is worth remarking that the global DS was the factor that better predicted this value (R² = 0.267; F = 18.592, p < 0.000), just like the group’s PR in attitude (R² = 0.077; F = 4.275, p < 0.044). In both contrasts, the global DS appeared as the only variable capable of meeting the entry criterion. The subjects’ attitude was equally appreciated by the class teachers and the support teachers, obtaining an identical average score of 2.54 points. Regarding the PR observed by the class teachers, its average reached 0.68 (over 1), while the PR observed by the support teachers reached 0.73. We show the contrast data in the table 5. Regarding the support teachers’ evaluation of the subjects’ attitude, the analysis showed the adjustment DS as the only variable in the equation, explaining 23.1% of the OV (F = 11.099, p < 0.002). In relation to the PR in attitude evaluated by the support teachers, not a single variable was introduced into the regression equation.

Table 5. Correlation between the evaluations made on attitude (A) and the values and scores obtained by the EG in the different measures contemplated

Class teacher’s (A) Class teacher’s (A) PR Support teacher’s (A) Support teacher’s (A) PR

Class teacher’s (A)

Class teacher’s (A) PR

Support teacher’s (A)

Support teacher’s (A) PR

EDPL DS

APSL Pretest

APSL Posttest

IS

0,397**

0,504**

0,106

0,164

0,090

---

0,443**

0,561**

0,139

0,187

0,276*

0,224

0,032

-0,228

0,443**

---

0,298

0,352*

0,474**

0,457**

0,314*

0,295

0,005

0,561**

0,298

---

0,495**

0,245

0,119

0,325*

0,169

-0,166

0,139

0,352*

0,495**

---

Adjustment DS

LM

0,345* 0,446** 0,391* 0,280

* p < 0.05; ** p < 0.01

3.4 Predictive value in relation to the IQ test Table 6. Correlation between the IQ scores and the values obtained by the EG in the different measures contemplated Meta DS

Global DS

Class teacher’s (AP)

Class teacher’s (PR)

Support teacher’s (AP)

Support teacher’s (PR)

IQ Pretest

IQ Posttest

IQ IS

IQ Pretest

---

0.720**

-0.275*

0.032

0.092

0.082

0.253

0.219

0.033

0.149

IQ Posttest

0.720**

---

0.469**

0.154

0.168

0.147

0.306*

0.367*

0.239

0.118

IQ IS

-0.275

0.469**

---

0.172

0.129

0.127

0.120

0.285

0.397*

0.026

Meta DS

0.032

0.154

0.172

---

0.972**

0.544**

0.434**

0.476**

0.301

0.195

Global DS

0.092

0.168

0.129

0.972**

---

0.576**

0.466**

0.479**

0.274

0.254

LM

* p < 0.05; ** p < 0.01

The stepwise regression analysis made to evaluate the weight of the various variables on the posttest scores showed, for the EG, an equation on which the pretest was introduced first, with a value of R² = 0.519 (F = 59.267; p < 0.000). The subsequent introduction of the IS caused a 0.481 increment 2 of the R , the resulting model explaining the OV by 100%. The combination of the IS and the two DS (metacognitive and global) explained 24.8% of the posttest, resulting again in a significant contrast model. On the other hand, the analyses made for the CG equally revealed a regression equation on 2 2 which the pretest appeared first (R = 0.691) while the IS was introduced later on, increasing the R by 0.309 (p < 0.000). We also analysed the predictive value of the DS on the AP in relation to the IQ scores. The resulting equation for the AP evaluated by the class teachers revealed that the global DS 2 was the only one to meet the entry criterion, with a value of R = 0.324 (F = 25.930, p < 0.000). An identical result was obtained on the analysis of the PR observed by the class teachers. The DS appeared as the only variable in the equation, explaining 24.1% of the variance. For the evaluation of the AP made by the support teachers, it was as well the global DS the one that better predicted the outcome, with a 24.6% of the OV. The metacognitive DS obtained similar correlation levels to those of the global DS and when the individual introduction method was used, the results of the analysis showed a similar percentage of explained variance, proving to be a significant contrast. On the other hand, the analysis made of the PR evaluated by the support teachers showed that the IS was the 2 better predictor, with a value of R = 0.158 (F = 6.732, p < 0.014). For the LM, no variables met the entry criterion stipulated. Stepwise regression analyses for the CG revealed that on any of the contrasts carried out according to the different evaluations made by class teachers and support teachers on the subjects’ AP and PR, as well as on their LM, neither of the variables contemplated was initially included in the regression equation. Generally speaking, the observed correlations between AP and the IQ scores were medium or low, even negative in certain cases.

4

DISCUSSION AND CONCLUSIONS

The present work gathers alternative proposals of DA that essentially include contextualised activities, closely related to school contents [29]; [10]; [23]; [15], and for which prediction is not the fundamental objective [3]; [4]; [5]; [6]. These proposals understand that the basic objective of DA must be the improvement of subjects and the optimisation of the educational process [9]; [11]. Predicting the behaviour or performance of the pupils would have sense only to the extent on which it could

contribute to the validation of an ecological assessment procedure and, together with it, of the information associated to it. The relevance of establishing the predictive value of a dynamic test would therefore be based in gathering data that allow to infer the action mechanisms of the test and to make proposals to optimise the learning process. In this sense, we have extracted from the experimental application process a series of methodological patterns of mediation that proved to be effective in the improvement of the evaluated processes, or in the gathering of valuable information for the optimisation of the teaching-learning process. The analysis of the mediation patterns [11] has given rise to the determination of those that are key elements to explain the observed improvements in the different tests. One of the action mechanisms related with the validity of the process and the results obtained, as well as with the predictive capacity of the dynamic tests, has to do precisely with the elaboration of specific tests for specific learning contexts, i.e., with the specificity of contents and their contextualisation [1]; [17]. The inclusion of activities typical of the school context into the dynamic tests could facilitate a greater connection between the results of the DA and those of school learning [4]; [17]; [9]. In this sense, our proposal contained activities for the various contexts related to the process of learning to read, among them: phonological conscience, the establishment of relations between texts and reading pseudowords [11]. In our opinion, the inclusion in the DA procedures for specific contents of these specific learning activities – which are usually worked and evaluated along the processes of school teaching and learning – is especially relevant in order to obtain predictive values that are both more consistent and ecologically valid [9];[16]. In relation to the predictive validity of the DA tests, in the introduction to this work we mentioned the difficulties encountered to establish relations between learning potential measures and AP. In this sense, it did not seem strange to us that the EDPL DS did not maintain high correlation levels with the posttest (which could be considered a performance measure of the contents in the area of Language) or with the marks. On the other hand, as we were able to prove, the two DS were the ones that better predicted the AP and the PR evaluated by the teachers. It is thus possible that in these performance evaluations other mechanisms may have acted that could have positively influenced the obtaining of significant correlation levels. In this sense, the fact that we facilitated the teachers certain qualitative evaluation items, together with the processes that had to be evaluated, could have contributed to the evaluations having a more process-like and less static character, thus explaining the positive and significant correlation results. In relation to the analysis of the predictive value regarding comprehension, we have shown the insufficiency of the pretest scores to explain the direction of the differences observed between the initial and the final phases of the criterion tests, and the performance of the subjects during the application of the dynamic device. On the contrary, the DS significantly predicted most of the values analysed. For the subjects’ AP and PR, the DS was the variable that better explained the scores obtained by the pupils, reaching significant values and being, in addition, the only variable included in the equation when stepwise regression analyses were implemented. From the results, we can conclude that the posttest and the IS explained most of the variance for the CG. On the contrary, the consideration of the DS in the EG seems to have significantly contributed to explain AP and PR. In relation to the gainers and non-gainers subgroups, the analyses equally revealed that the DS explained better than the pretest the evaluation of their AP. On the other hand, the analyses made to assess the predictive value regarding the APSL scale revealed a strong connection between the global DS and the adjustment DS. These data are relevant, not only concerning the degree of consistency of the dynamic device, but also in reference to the close relation we have found between the subjects’ general performance in the device sessions and the application of abilities related with adjustment processes. The two DS showed a predictive value of the AP that was significantly higher than that of the scores obtained in the APSL, both the initial and post-treatment ones. In this sense, for the CG, the absence of treatment that the application of the EDPL entailed would have validated the initial prediction of the pretest on AP. In the EG, on the contrary, the application of the EDPL significantly contributed to explaining AP. This prediction value of the two DS was also manifested in relation to the subjects’ attitude before reading, also evaluated by the teachers. Regression analyses clearly showed that – while for the CG the pretest was the only variable significantly explaining AP and PR – the regression equation of the EG only contemplated the two DS (global and adjustment), significantly explaining both variables performance and progress. In relation to the analysis of the predictive value of the EDPL on cognitive performance, the results showed that, in the EG, the IS could explain posttest scores very significantly. In the EG, the results revealed a greater weight of the scores that reflected the change produced in the subjects. In effect, while for the CG the pretest explained 69.1% of the posttest in Cattell, for the EG the pretest only explained 51.9%. In this sense, the variance percentage explained by this group’s improvement (48.1%) revealed the essential contribution of the applied treatment. Likewise, the correlation values of the EDPL DS with the IS in Cattell were higher than the ones obtained by the pretest scores. On the other hand, we were

especially interested in determining what scoring, static or dynamic, had a greater relation with AP [30]. Although high levels of correlation between IQ and AP have been evidenced [2]; [25], one of our hypotheses set out that a DA based on the learning process, and contextualised in relation to the evaluation contents, would be a more important explanatory factor than the information obtained from a test’s static application, especially if the test was uncontextualised. The results neatly showed that the correlation levels obtained by the DS in relation to AP and PR were significantly higher than those observed for the IQ scores. Our proposal is based on those dimensions considered more relevant by Carlson and Wiedl [20] in the application of the DA processes, as well as in their validation. These authors highlight, on the one hand, those general cognitive structures that basically refer to self-regulation processes and that are closely related with the achievements of the specific cognitive structures – the name given to specific domains and functions like reading. In order to optimise the efforts focused on the implementation of the DA processes, these researchers described three necessary steps [20]: (1) to establish, both theoretically and empirically, the relation between general cognitive structures and specific domains and functions, (2) to demonstrate the predictive validity of the dynamic tests on specific domains, and (3) to develop educational intervention processes based on the theory of the general cognitive structures – like psychological instruments of mediation – and determine the greater predictive validity of dynamic tests when compared to that of traditional evaluative approaches. In brief, the aim of these researchers was to establish those areas of cognitive functioning that were most relevant, from a theoretical and empirical point of view, for the explanation of specific domains; these would be the areas on which DA validation would make more sense. In our case, the analysis of the theoretical and methodological bases, and of the process and the results of our research, offers us the possibility of establishing theoretical and empirical links between general cognitive structures and specific domains. And this is possible due to the fact that our results show the relation between the application of selfregulation abilities and the performance of specific functions (through the teachers’ evaluation), including text comprehension. Likewise, we have showed, through the analysis of the results, that the EDPL scores very significantly predicted the subjects’ performance in reading. In addition, our device contemplates the importance of mediating the use of metacognitive processes in the development of learning strategies. Finally, we have determined the greater predictive value of dynamic scores as compared to the static evaluation approaches to reading, and to the evaluation of intelligence.

REFERENCES [1] Kozulin, A. & Garb, E. (2001). Dynamic Assessment of EFL Text Comprehension of At-Risk th Students. Paper presented at the 9 Conference of the European Association for Research on Learning and Instruction. Fribourg, Switzerland, 2001. [2] Sternberg, R.J. & Grigorenko, E.L. (2003). Evaluación dinámica. Naturaleza y mediación del potencial de aprendizaje. Barcelona: Paidós. [3] Delclos, V.R.; Vye, N.J.; Burns, M.S.; Bransford, J.D. & Hasselbring, T.S. (1992). Improving the quality of instruction: Roles for dynamic testing. En H.C. Haywood y D. Tzuriel (Eds.), Interactive testing. New York: Springer-Verlag, pp. 317-332. [4] Ruijssenaars, A.J.J.M.; Castelijns, J.H.M. & Hamers, J.H.M. (1993). The validity of learning potential tests. En J.H.M. Hamers; K. Sijtsma & A.J.J.M. Ruijssenaars (Eds.), Learning potential testing. Amsterdam: Swets and Zeitlinger, pp. 69-82. [5] Jensen, M.R. (2000). The MindLadder model: Using dynamic assessment to help students learn to assemble and use knowledge. En C.S. Lidz & J.G. Elliott (Eds.), Dynamic Assessment: Prevailing models and applications, Greenwich, CT, Elsevier-JAI, pp. 187-227. [6] Elliott, J. (2003). Dynamic assessment in educational settings: realising potential. Educational Review, 55, 1, 15-32. [7] Lidz, C.S. & Elliott, J.G. (Eds.) (2000): Dynamic assessment: Prevailing models and Applications, Greenwich, CT, Elsevier-JAI. [8] Lauchlan, F. & Elliott, J. (2001). The Psychological Assessment of Learning Potential. British Journal of Educational Psychology, 71, 647-665. [9] Schlee, J. (1985). Förderdiagnostik–Eine bessere Konzeption? En R.S. Jäger; R. Horn y K. Ingenkamp (Eds.), Tests and trends, Weinheim, Beltz, pp. 82-208. [10] Wells, G. (2001). Indagación dialógica. Hacia una teoría y una práctica socioculturales de la educación. Barcelona: Paidós.

[11] Navarro, J.J. (2008). Evaluación dinámica de procesos implicados en la lectura. Tesis doctoral. Universidad de Sevilla. [12] Lidz, C.S. (2004). Comment on Curriculum-based Measurement: Describing Competence, Enhancing Outcomes, Evaluating Treatment Effects, and Identifying Treatment Nonresponders by Fuchs and Fuchs, Journal of Cognitive Education and Psychology, 4, 1, 131-133. [13] Guterman, E. (2002). Toward dynamic assessment of reading: applying metacognitive awareness guidance to reading assessment tasks, Journal of Research in reading, 25, 3, 283-298 [14] Kalyuga, S. & Sweller, J. (2005). Rapid Dynamic Assessment of Expertise to Improve the Efficiency of Adaptive E-learning, ETR&D, 53 (3), 83-93. [15] Haywood, H.C. & Lidz, C.S. (2006). Dynamic Assessment in Educational Settings, En: H.C. Haywood & C.S. Lidz, Dynamic assessment in practice: clinical and educational applications. Edit. Cambridge. [16] Thurman, S. & McGrath, M. (2008). Environmentally Based Assessment Practices: Viable Alternatives to Standardized Assessment for Assessing Emergent Literacy Skills in Young Children. Reading & Writing Quarterly, 24, 1, 7-24. [17] Hamers, J.H.M.; Pennings, A. & Guthke, J. (1994): Training-based assessment of school achievement. Learning and Instruction, 4, pp. 347-360. [18] Lidz, C.S.; Jepsen, R.H. & Miller, M.B. (1997): Relationships between cognitive processes and academic achievement: Application of a group dynamic assessment procedure with multiply handicapped adolescents. Educational and Child Psychology, 14, 56-67. [19] Naglieri, J. & Das, J. (1987): Construct and criterion related validity of planning, simultaneous and successive cognitive processing tasks. Journal of Psychoeducational Assessment, 4, 353-363. [20] Carlson, J.S. & Wiedl, K.H. (2000): The validity of dynamic assessment. En C.S. Lidz & J.G. Elliot (Eds.), Dynamic Assessment: Prevailing Models and Applications (6), pp. 681-712. Greenwich, CT, Elsevier-JAI. [21] Lidz, C.S. (1991): Practitioner’s guide to dynamic assessment. New York: Guilford. [22] Calero, M.D. (2004). Validez de la evaluación del potencial de aprendizaje. Psicothema, 16, 2, pp. 217-221. [23] Lidz, C.S. (2000). The application of Cognitive Functions Scale (ACFS): An example of curriculum-based dynamic assessment. En C.S. Lidz & J.G. Elliott (Eds.), Dynamic Assessment: Prevailing models and applications, Greenwich, CT, Elsevier-JAI, pp. 407-439. [24] Saldaña, D.; Mora, J. & Moreno, J. (2007). Evaluación dinámica en retraso mental severo: descripción de un procedimiento ecológico para la valoración de la metacognición. En J.N. García (Coord.), Dificultades del Desarrollo, Madrid: Edit. Pirámide. [25] Cattell, R.B. & Cattell, A.K.S. (1974/2001): Test de factor «G» (Escalas 2 y 3). Madrid: TEA. [26] Budoff, M. (1987). The validity of learning potential assessment. En C.S. Lidz (Ed.), Dynamic assessment: An interactional approach to evaluating learning potential. New York: Guilford Press, pp. 52-81. [27] Resing, W.C.M. (2000). Assessing the learning potential for inductive reasoning (LIR) in young children. En C.S. Lidz y J.G. Elliott (Eds.), Dynamic assessment: Prevailing models and applications. Greenwich, CT: Elsevier-JAI, pp. 224-262. [28] Botella, J.; León, O.; San Martín, R. & Barriopedro, M.I. (2001): Análisis de Datos en Psicología I. Madrid: Pirámide. [29] Cioffi, G. & Carney, J. (1983). Dynamic assessment of reading disabilities. Reading Teacher, 36, 764-768. [30] Swanson, H.L. & Howard, C.B. (2005). Children with Reading Disabilities: Does Dynamic Assessment Help in the Classification? Learning Disability Quarterly, 28, 1, pp. 17-34.