Understanding learning difficulties—A predictive

Studies in Higher Education

ISSN: 0307-5079 (Print) 1470-174X (Online) Journal homepage: http://www.tandfonline.com/loi/cshe20

Understanding learning difficulties—A predictive research model A.H. Johnstone & H. El-Banna To cite this article: A.H. Johnstone & H. El-Banna (1989) Understanding learning difficulties—A predictive research model, Studies in Higher Education, 14:2, 159-168, DOI: 10.1080/03075078912331377486 To link to this article: http://dx.doi.org/10.1080/03075078912331377486

Published online: 05 Aug 2006.

Submit your article to this journal

Article views: 154

View related articles

Citing articles: 23 View citing articles

Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=cshe20 Download by: [University of Glasgow]

Date: 05 July 2016, At: 06:10

Volume 14, No. 2, 1989

Studies in Higher Education

159

Understanding Learning Difficulties a predictive research model A. H. JOHNSTONE & H. EL-BANNA

Downloaded by [University of Glasgow] at 06:10 05 July 2016

Science Education Research Unit, University of Glasgow

ABSXRACT This paper arises from investigations of the topics in science which students

perceive to be difficult. An information processing model is described which relates student performance to the amount of information to be processed in a learning or problem-solving situation. When the student working memory capacity is exceeded, there is a sharp drop in performance, but some students (~10% ) continue to operate efficiently with problems which exceed their capacity; they are probably employing chunking devices that enable them to reduce the problem demand to less than their limit of capacity. Empirical measurements give support to the model and suggest hypotheses for future research.

In the 1970s (Duncan & Johnstone, 1974) a detailed analysis was done of the content of parts of school chemistry about which pupils and teachers had complained as being too difficult. Part of the approach to the problem consisted of setting test questions on subsections to try to isolate the trouble spot or spots. The results were surprising and, at the time, inexplicable. A cluster of questions was found in which pupils had a high success rate ( > 0 . 6 ) and another cluster appeared in which there was a poor success rate ( < 0 . 3 ) but there was a gap in between these clusters. There were no questions which were generating a middling success rate between 0.3 + and 0.6 (Fig. 1). A broadly normal distribution might have been expected, but in fact it did not emerge. Other workers prodding into other areas of the syllabus which pupils and teachers perceived to be difficult came up with similar performance distributions--a 'hole in the middle'.

10-

~6 5 6

Z

0.0

0.1

0.2

0.3

I 0.4

I 0.5

0.6

0,7

0.8

0.9

1.0

Facility

FIG. 1. Number of questions in each band of facility (facility is the proportion of the sample getting a question correct).

160 A. H. Johnstone & H. El-Banna

Downloaded by [University of Glasgow] at 06:10 05 July 2016

It was not until 1980 that a tentative working hypothesis was put forward in terms of information processing theory (Johnstone & Kellett, 1980). In our most recent study we re-examined this earlier work and added to it about 100 questions from the Scottish Examination Board based upon these difficult topics (Johnstone & E1-Banna, 1986). This time, however, we approached the problem in a different way. Each question was examined by a jury for its demand (Z). By demand we mean the maximum

number of thought steps and processes which had to be activated by the least able, but ultimately successful candidate in the light of what had been taught. The procedure was to choose at random 25 scripts where the students had scored full marks in a given question and to examine their routes. These were largely numerical questions which lent themselves to a stepwise analysis in terms of what had been recalled, transformed, deduced and concluded. Each question was treated in the same way with another random sample of 25 successful students. Jury members then compared the results of their analyses for each question and agreement was obtained (if need be by majority) for the number of 'steps' in the longest route for each question (the Appendix shows a typical analysis). This was a purely operational method for assessing demand because the maximum demand on working space is likely to take place before any sequencing is achieved. In fact our method may be an underestimate of demand. It is certainly no stronger than an estimate, but it does enable us to arrive at a relative indication of demand. This question analysis was carried out by researchers who had no hand in setting or marking the questions. The facility values were then plotted against 'demand', in the expectation of obtaining a negative correlation. Such a correlation was obtained, but not in the form expected (Fig. 2) (this figure has been simplified by choosing every fourth question from the sample for plotting). We obtained an S-shaped curve indicating a high and a low plateau with a rapid drop between them. Here again was the 'hole in the middle' which we had seen long before but could not interpret. The shape of the curve was reminiscent of curves describing phenomena in which something is taken to a limit beyond which there is a rapid change. The vertical part of the curve fell on the demand axis at between 5 and 6, setting us off to reread Miller's work of the 1950s (Miller, 1956). Perhaps the curve was showing an overload in processing (working memory) capacity and that the areas of difficulty were those where this breakdown was likely to occur either because of the nature of the science or because of the way in which it was being taught and learned or some blend of these. Since these results were obtained from a composite group of some 20,000 candidates, it was clear that, if the 'capacity' of individuals varied (as shown by Miller), some of the curve's peculiarities could be explained. The upper plateau must consist of questions which were within the working memory capacity of all, but carelessness, forgetfulness or lack of interest could depress the achievement. The points on the steep part of the curve could indicate questions which were now proving too much for some, but were still within the compass of the rest. The lower plateau represented the questions which had now gone beyond 90% of the sample. The remaining successful 10% had something to teach us. To test some of these propositions our attention turned to a sample of some 350 tertiary students in their first year of a science degree who were reading chemistry. We employed two tests, not of short-term memory, but of the capacity to hold pieces of information and operate upon them. This was an attempt to quantify the shared holding/thinking space which an individual might possess (X). The definition of 'piece of information' is a very woolly one especially if a subiect is operating in a familiar situation. He can bring into play all sorts of chunking devices by which many pieces of information can be handled as if they were one. A 10-digit telephone number would be 10 pieces of information for one person but many fewer

Understanding Learning Difficulties 161 1.0 \ o\

0.9

~

•

•

•

\

•

•

\. 0,8

\ 0.7

\

\

\

\ \

0.6

\ >

Downloaded by [University of Glasgow] at 06:10 05 July 2016

ii

0.5

\

\

\ \ \

0.4 •

\

0.3

0.2

0.1

\

\ \

0

I 1

I 2

I 3

i 4

I

I

I

I

J

[

5

6

7

8

9

10

L

\1

l

11 12 13

No. of thought steps

FIG. 2. A plot of facility value in objectivechemistryquestions versus the number of thought steps needed to solve the questions. for another who recognised the system and used it. 0413398855 is broken in the directory as 041-339-8855. 041 is the 'chunk' for Glasgow; 339 is the district and 8855 is the subscriber's number. T o avoid such chunking, tests had to be used which were in areas unfamiliar to the students. T h e tests were 'digit span backwards' (in which a sequence of numbers had to be memorised and then turned over and given back in reverse order) and the Figural Intersection Test (Pascual-Leone, 1974). The agreement between the tests is shown in Table I). For the purposes of the set of experiments described below we confined ourselves to the subset with identical scores, but in subsequent work we have taken in those in the +_ 1 category. TABLEI. Agreementbetween tests Performance on two tests Identical score Difference -+1 Difference > -+1 Others Total

No. 271 (74.5%) 40 23 30 364

162

A. H. Johnstone & H. El-Banna

The sample was distributed, by 'capacity', as shown in Table II. These students were then followed for a year and their performance on three university examinations was recorded. These examinations were set, and marked independently of the researchers. The 'demand' of each question was determined by a jury in consultation with the setters.

Downloaded by [University of Glasgow] at 06:10 05 July 2016

TABLE II. Distribution according to capacity Capacity (X)

No.

5 6 7

98 110 63

Total

27t

The model on which our hypotheses were raised was as follows. A necessary (but not sufficien 0 condition for a student to be successful in a question is that the demand of the question should not exceed the working memory capacity of the student. If the capacity is exceeded, the student's performance should fall unless he has some strategy which enables him to structure the question and to bring it within his capacity. An ideatised set of curves would look like Fig. 3. The actual experimental results are shown for one examination in Fig. 4. I f we take 50% as a marker, it can be seen that the capacity 5 students drop from a facility of 0.6 to less than 0.4 as the demand rises from 5 to 6. The capacity 6 students fall from 0.6 to 0.2 as the demand rises from 6 to 7 and the capacity 7 students fall from 0.7 to 0.3 on moving demand from 7 to 8. There is a clear similarity between Figs. 3 and 4. 1.0

X=5

" i

X=6

~

'~"-'-

> u_

X= 7

0.5

0

I 1

I 2

I, 3

I 4

It

5

' 6

i

f~ 8

I 9

FIG. 3. Expected performance in chemistry of students of differing X space in questions of different Z demand.

In another of the examinations there were no questions of demand 6. We therefore predicted that the students of capacity 5 and 6 would perform similarly since there was no

Understanding Learning Difficulties 163 1.0 0.9 0.8 o 0.7 0.6

k Downloaded by [University of Glasgow] at 06:10 05 July 2016

>ii

0.5

0,4

\k~\x 0.3

\

\\\

Ok.

×\\\\

\. \

0,2 \\..

X

"\

0.1

__.%

I

I

J

I

I

I

I

I

3

4

5

6

7

8

9

10

Z

FIG. 4. C o m p a r i s o n (O--O) X=5.

of the average

FV

f o r all g r o u p s

of different

X space.

(C)-.-C)) X=7;

x--- x)

X=6;

question of demand 6 which would allow the capacity 6 students t o show their superiority over the capacity 5 group. The results for this exam are shown in Fig. 5. In Figs. 4 and 5 a 'nesting' effect is seen in that the 7 curve is above the 6 curve and it is above the 5 curve at all points. Even when a question is within the capacity of all the students, the capacity 7 students on average outperform those of lower capacity. There is also a very heartening feature in that when their capacity is exceeded, about 10% of students can continue to function successfully. In other words, strategies can be learnt which enable a learner to overcome capacity limitations. One might describe a person who is successful in any discipline as one who has developed a good set of strategies in that subject and who has learned (consciously or otherwise) to operate well outside his capacity. This also raises the question about how to improve the performance of the other 90%. Can strategies be taught? This is the subject of another of our studies not reported in this paper. If what we have reported so far has any substance, we should be able to detect it in gross phenomena such as all-over pass rates. If a question paper has questions of different demands, only those questions with a demand less than 6 will be accessible to those of capacity 5. Those of higher capacities will have access to more of the paper. This should be reflected in the results (Table III). Those students who have done well in the term exams are exempt from the finals. Those who are not exempt in the finals have two opportunities to sit the final exams (June and September). The relationship between capacity and success is very clear.

164

A. H. Johnstone & H. El-Banna 1.0-

0.9

0.8

0.7

\

\

0.6 > LL

\

\

\'~

~\\

~

Downloaded by [University of Glasgow] at 06:10 05 July 2016

0.5

\ ~\\

0.4

\ ~

~\\

0.,~

\~

\ ~\\\

0.2

",, "\.

0.1

O ~

I 3

I 4

I 5

I 6

I 7

I 8

I 9

Z

FIG. 5. Comparison of the results from university students of different X space on questions of different Z demand (FV). ( O - - - O ) X=7; ( x - - - x ) X=6; ( O - - O ) X=5.

TABLEIII. Results of the sample according to capacity (%) Capacity

Exempt

Pass June

Pass Sept.

Fail

5 6 7

11.2 30.9 63.5

41.8 52.7 31.7

17.4 7.3 4.8

29.6 9.1 0.0

Since the second and succeeding years are made up of those who are successful in the first year, the composition of each successive year must become richer in capacity 7 and poorer in capacity 5 students. This is, in fact, borne out experimentally (Table IV).

TABLE IV. Results for subsequent years (%) Capacity

1st year

2nd year

3rd year

5 6 7

36 41 23

27 39 35

18 40 42

Understanding Learning Difficulties 165 Discussion

All that has been reported in this paper has been subjected to a series of replication studies with very similar results. Composite graphs generated from studies on nine chemistry examinations are shown in Figs. 6, 7 and 8. Each point through which the graphs are drawn is the average facility score for all questions of the same demand. The range of the facility scores is shown by the range bars through each point. 1.0

0.9

Downloaded by [University of Glasgow] at 06:10 05 July 2016

0,8

0.7

0,6

> LL

0.5

0,4

0.3

0.2

0.1 -0

o -~,

I 3

I 4

I 5

I 6

I 7

I 8

J 9

I

10

Z

FIG. 6. Overall results from X = 5 university students (two years of testing).

The length of these range bars indicates to some extent the uncertainty in the estimate of the demand of each question. Where no range bars occur, there is either only one question of that demand or the questions of the same demand had the same facility. The general Sshaped patterns are maintained. It would be unwise to suggest that all problems of learning and testing can be explained by this simple model, but it does suggest a mechanism for some of the problems which do exist and a mechanism by which they might be overcome. Our colleagues are now exploring the insights which the model gives for teaching and learning of all kinds. So far very promising results are emerging in the areas of testing, laboratory work, computer displays, worksheets, textbooks and university tutorials. The teacher can control the amount of useful information (the signal) which the learner has to process and can also limit, or even exclude, the extraneous, distracting information (the noise) in a learning situation. During the early learning period the 'signal' to 'noise' ratio can be carefully controlled and then the 'noise' can be increased to give students experience of distinguishing one from the other. Students can

166

A. H. Johnstone & H. El-Banna 0.9

0.8

0.7

0.6 > u.

0.5

Downloaded by [University of Glasgow] at 06:10 05 July 2016

0.4

0,3

0.2

0,1

J~

I

t

I

L

I

1

]

I

3

4

5

6

7

8

9

t0

z

FIG. 7. Overall results from X = 6 university students (two years of testing). 0.9,

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

o

---"v

I

I

.,[

~

I

I

I

I

3

4

5

6 Z

7

8

9

10

FIG. 8. Overall results from X = 7 university students (two years of testing).

Understanding Learning Difficulties

167

be taught chunking devices by which they can chop problems down to size to fit their capacity. Often teachers do not 'hear' the 'noise' because they have long since got used to it, but for first-time learners, the problem is acute. It may be that the strategies devised by the 10% of students who perform outside their capacity can be transmitted to their peers. Our adult strategies may be inappropriate to a novice. This model has predictive power and should enable us to raise and test important hypotheses in the future in many subject areas.

Downloaded by [University of Glasgow] at 06:10 05 July 2016

Correspondence: Dr A. H. Johnstone, Reader in Chemical Education, Department of Chemistry, University of Glasgow, Glasgow G12 8QQ, United Kingdom.

REFERENCES DUNCAN,I.M. & JOHNSTONE,A.H. (1974) Chemistry Check-up (London, Heinemann). JOHNSTONE,A.H. & EL-BANNA,H. (1986) Capacities, demands and processes--a predictive model for science education, Education in Chemistry, 23, pp. 80-84. JOHNSTONE, A.H. & KELLETT, N.C. (1980) Learning difficulties in school science--towards a working hypothesis, European Journal of Science Education, 2, pp. 175-181. MILLER, G.A. (1956) The magical number seven, plus or minus two: some limits on our capacity for processing information, PsychologicalReview, 63, pp. 81-97. PASCtJAL-LEONE,J. (1974) Manualfor FiguralIntersection Test (York University,Canada).

Appendix 1 Demand can be thought of as being generated by (i) the information in the question which has to be processed; (ii) the information which has to be recalled to add to the given information; and (iii) the processes which have to be activated to deal with the information; processes such as deduction, transformation and calculation.

Example 'What volume of molar hydrochloric acid would be exactly neutralised by ten grams of chalk?' Demand: 1. 2. 3. 4. 5. 6. 7. 8. 9.

chalk---calcium carbonate (recall) calcium carbonate =-CaCO 3 (recall or deduce) Formula weight of CaCO3= 100 g (calculate) When it reacts with hydrochloric acid, what are the products? (recall) Write a balanced equation (transformation) Recognise that 1 mole C a C O 3 ~ 2 moles HC1 (deduce) -= 2 litres of molar HC1 (recall) 10 g CaCO3 ~ 1/10 mole ~ 1/5 mole HC1 (deduce) ~ 1/5 litre molar HC1 (recall) = 200 ml molar HC1

168

A. H. Johnstone & H. El-Banna

This was the longest successful route taken by the students and so was given a demand number of 9. Experienced teachers were able to do this in four steps by a series of chunking devices. 1. 10 g chalk~ 1/10 mole CaCO 3 (recall) 2. Chalk versus HC1 is a 1:2 ratio reaction (recaI1) 3. 1/10 CaCO3~ 1/5 mole HC1 (deduce) 4. ------200ml molar HC1 (deduce)

Downloaded by [University of Glasgow] at 06:10 05 July 2016

The demand must be related to the stage at which the respondents are in their learning. In this case the demand was taken to be 9 for O level students and 4 for experienced teachers.