PISA, and could have been used to describe mathematics items in general. 48. Northern Lights on PISA 2003 northern.qxp20 04/05/06 9:51 Side 48 ...
northern.qxp20
04/05/06
9:51
Side 47
Chapter 4
What are the Characteristics of the Nordic Profile in Mathematical Literacy? Rolf V. Olsen and Liv Sissel Grønmo
Abstract The previous chapter dealt with the development of the relative achievement profiles for countries across all the items within the domain of mathematical literacy in PISA 2003. Analyses of the overall pattern in these profiles indicated that the Nordic countries’ profiles were relatively strongly correlated. Furthermore, the analysis also revealed that other groups of countries with similar relative achievement profiles existed. In this chapter these clusters are further analysed by studying characteristics of the items. The findings reveal that the profiles of the Nordic and the English-speaking countries are mainly accounted for by variables describing what could be termed ‘realistic mathematics’. This finding is discussed in relation to curricular approaches to mathematics competency and learning in the Nordic countries.
Nordic abstract Det forrige kapitlet handlet om hvordan man kan utvikle såkalte relative prestasjonsprofiler på tvers av alle oppgavene i matematikk for hvert land. Analysene av disse profilene viste at de nordiske landene i noen grad presterer relativt godt eller dårlig for de samme oppgavene. Analysene viste også at det finnes tilsvarende grad av samsvar mellom profilene for flere andre grupper av land. I dette kapitlet søker vi å forstå hva som karakteriserer de relative prestasjonsprofilene for de nordiske landene – både som én gruppe og hver for seg. Dette gjøres ved først å karakterisere hver enkelt oppgave ved hjelp av ulike klassifiseringer. Det overordnete funnet er at de nordiske landene presterer relativt sterkest på oppgaver som tester det vi kaller ’realistisk matematikk’. Resultatene diskuteres i lys av hva som har vært fremtredende mål for grunnskolematematikken i disse landene.
47
northern.qxp20
04/05/06
9:51
Side 48
Northern Lights on PISA 2003
Introduction Identifying possible explanations for why groups of countries have similar relative achievement profiles across mathematics items, as presented in the previous chapter, is a challenging task. Given the obvious identity of several of these clusters, or the ease with which we could label them, it is of course tempting to jump to conclusions about cultural antecedents and systemic factors that could explain the background characteristics of these groups of countries. However, it may not be so easy to derive these conclusions from the data themselves. There are few, if any, variables in the PISA data that can be used to describe the historical, social, political, economical or cultural factors that could be seen as background influences leading to links between countries in mathematics achievement profiles across several items. The most direct approach to describing the profiles for the clusters of countries using the data available is to identify more precisely the particularities of the items defining each of the profiles. This may be done in several ways. One possibility would be to first identify a smaller group of key or defining items, in the relative achievement profiles. These items could then be investigated in detail in order to develop verbal descriptions of the profiles that go beyond the very specific context of the items. Alternatively, one could start by describing the items in more abstract or general terms, and then apply these descriptors to all the items. Of the two alternatives the latter is preferable since this approach utilises all the items in the pool. Thus, this approach has the potential to identify key aspects of the achievement profiles of the clusters that are largely independent of the actual items in the pool.
Method The item pool in mathematics consists of 84 items. Some of the items refer to a stimulus material of some length, some don’t. Some items ask the student to formulate their own answers, while some ask students to select the most appropriate from several given responses. Some of the items refer to phenomena related to everyday life, while others refer to scientific phenomena. And so on, the point being that it is possible to develop a great range of different descriptors characterising the items, and in some cases the item characteristics may be present to varying degrees. Some of the item descriptors we have chosen to develop are directly based on the PISA framework (OECD, 2003), while others were developed independently. The framework categories are accounted for in more detail in chapter 2 by Kupari and Törnroos. The descriptors are in principle independent of the actual items present in PISA, and could have been used to describe mathematics items in general. 48
northern.qxp20
04/05/06
9:51
Side 49
Chapter 4: What are the Characteristics of the Nordic Profile in Mathematical Literacy?
Table 1 Description of some broad descriptors used to classify the mathematics items Variable name Item format
Description [with variable values in brackets] Classification of items into the two main formats, constructed responses [1] or selected responses [0].
Context four Context two
The mathematics items in PISA are classified in the framework into four different situations (personal [0], educational/occupational [1], public [2] or scientific/intra-mathematical [3]) which may be perceived as ordered according to the distance from the situation to the student (‘Context four’). The context may also be classified dichotomously by separating the scientific context [1] from the other contexts set in a more real-life context [0] since this is a purer context where the mathematical aspects themselves are central (‘Context two’).
S&S C &R Quantity Uncertainty
The mathematics items are classified in the framework according to four phenomenological topics; Space and Shape (‘S & S’), Change and Relationship (‘C & R’), Quantity and Uncertainty. The four variables suggested classify the items as belonging to each of the topics or not.
Competency
The items are classified in the framework according to the main competency involved. The three competencies range from what has been termed the ‘reproduction’ cluster items [0] to the ‘reflection’ cluster items [2]. In between these two extremes is a group of items classified as the ‘connections’ cluster [1].
p-value
The overall international average p-value of the items.
RealMath
The degree to which an item confronts the students with a realistic problem relating to their personal lives or to citizenship.
Algebra
Classifies the items that include an explicit algebraic expression versus those not including algebraic expressions.
Calculations
Classifies the items that to a large degree require calculations.
Graphics
Classifies the items that include graphical representations of quantities.
Tables
Classifies the items that include representations of quantities in tables.
Non-continuous
Classifies the items that include information presented in a noncontinuous way, e.g. graphs and tables, but also sketches or illustrations other than graphs
Complex reading
Classifies the items that mainly require the student to handle the information given, for instance by sorting relevant from irrelevant.
49
northern.qxp20
04/05/06
9:51
Side 50
Northern Lights on PISA 2003
However, we have selected the descriptors that we initially perceived to be of particular relevance for the PISA study and for this particular item pool. A useful descriptor would be one that may be formulated so that independent evaluators of the items would agree whether an item had the characteristic or not, and furthermore, a useful item descriptor would divide the items into groups or assign a value to the items that would vary across them. Many of the descriptors in Table 1 are very straightforward and people would easily agree whether an item has the characteristics or not. The descriptor labelled as RealMath has however been constructed in a more complex manner. The two authors of this chapter started by each independently evaluating whether the items were dealing with ‘realistic mathematics‘, that is, mathematics perceived to be of particular relevance for everyday life or for life as a citizen. The definition of mathematical literacy in PISA highlights this aspect of mathematics, but nevertheless, the items in the pool refer to ‘real life’ mathematics to a varying degree. Even though according to the framework of the study all items in PISA relate to mathematics as students are supposed to need it in their daily or civic lives, the degree of relevance differs. In the two independent evaluations it became clear that the two authors had applied slightly differing definitions or criteria for evaluating whether the items dealt with mathematics in real life settings, ‘real life‘ referring to an item dealing with a mathematical problem or competency that is likely to be relevant to all citizens at one time or another, as opposed to items that are only likely to be relevant under very specific conditions. Even if we reconciled our definitions there were discrepancies in our evaluations of the items regarding this characteristic. One of the authors assigned a single value to each of the items in a holistic manner. The other author evaluated three slightly different aspects of the authenticity of the items: 1) whether or not the stimulus material presented had been extracted from an authentic text; 2) whether or not the item related to a realistic problem in the context supplied by the stimulus material; and 3) whether or not the underlying competency tested in the item could be considered highly relevant to ‘realistic mathematics’. Except for the variable categorising the authenticity of the stimulus material, all the variables describing how realistic the items were, were highly positively correlated to each other. It was therefore decided to establish the construct RealMath as the sum of the holistic evaluation system developed by one of the authors and the system for evaluating the realism of the problem and competency components of the item developed by the other author. Constructed in this way, the RealMath variable represents ‘realistic mathematics’ as a broader concept than can be represented by a single dichotomous variable. The coefficient alpha for the construct was close to 0.8. This number reflects the degree of consistency with which the two authors evaluated items as being ‘realistic’ or not. Examples of items that were categorised as RealMath are ‘Robberies’ and ‘Internet Relay Chat’, examples of items that were 50
northern.qxp20
04/05/06
9:51
Side 51
Chapter 4: What are the Characteristics of the Nordic Profile in Mathematical Literacy?
Table 2 Correlations between average cluster residuals and the broad item descriptors. The significant coefficients (p
