New Zealand teachers' conceptions of the purpose of

The complexity of teachers’ conceptions of assessment

1

THE COMPLEXITY OF TEACHERS’ CONCEPTIONS OF ASSESSMENT: TENSIONS BETWEEN THE NEEDS OF SCHOOLS AND STUDENTS

Lois R. Harris University of Auckland Gavin T. L. Brown Hong Kong Institute of Education

Running head: The complexity of teachers’ conceptions of assessment

Correspondence should be addressed to Dr Lois R Harris, School of Teaching, Learning, & Development, Faculty of Education, The University of Auckland, Private Bag 92019, Auckland, New Zealand or by email to [email protected]

Acknowledgements. This study was completed in part through the financial support of a University of Auckland Faculty of Education Faculty Research Development Fund grant (#3609273/9215) to the Measuring Teachers’ Assessment Practices project. The cooperation and participation of the teachers and schools in the MTAP project is appreciated. An earlier version of this paper was presented at the Australian Association for Research in Education (AARE) Annual Conference December 2008, Brisbane, Australia.

1


2

THE COMPLEXITY OF TEACHERS’ CONCEPTIONS OF ASSESSMENT: TENSIONS BETWEEN THE NEEDS OF SCHOOLS AND STUDENTS Abstract Teachers’ conceptions of assessment are important as they shape their usage of assessment practices. This study used a phenomenographic approach to examine the various purposes a sample of 26 New Zealand teachers ascribed to assessment. Seven purposes were discussed: compliance, external reporting, reporting to parents, extrinsically motivating students, organising group instruction, teacher use for individualising learning, and joint teacher-student use for individualising learning. This study showed that teachers hold complex conceptions of assessment and described using different assessments for differing purposes. It highlighted how teachers must consider divergent stakeholder interests when selecting assessments for students, balancing the needs of the society, the school, and the pupil. The data emphasised the particularly strong tension between what teachers feel is best for students versus what is deemed necessary for school accountability.

2


3

THE COMPLEXITY OF TEACHERS’ CONCEPTIONS OF ASSESSMENT: TENSIONS BETWEEN THE NEEDS OF SCHOOLS AND STUDENTS Introduction How teachers think about curriculum, subject matter, teaching, and learning has been shown to influence classroom practices (Clark & Peterson, 1986; Thompson, 1992) which in turn have consequences for student outcomes (Muijs & Reynolds, 2005). Assessment practices are particularly powerful as they have the capacity to facilitate or hinder student learning (Black & Wiliam, 1998). As beliefs and reasons underpin human behavior (Ajzen, 2005), teacher conceptions of assessment are important as they shape teacher use of assessment. Debates over best practice in assessment stem partly from divergent ideas about the purposes of assessment. There appear to be four major conceptions about the purpose of assessment discussed in research literature (e.g. Brown, 2008; Heaton, 1975; Shohamy, 2001; Torrance & Pryor, 1998; Warren & Nisbet, 1999; Webb, 1992). These are: •

Assessment improves teaching and learning (Improvement).

•

Assessment makes students accountable for learning (Student accountability)

•

Assessment demonstrates the quality of schools and teachers (School accountability)

•

Assessment should be rejected because it is invalid, irrelevant, and negative (Irrelevance)

The premise of the improvement conception is that assessment should enhance student learning and teaching quality (Black, Harrison, Lee, Marshall, & Wiliam, 2003; Black & Wiliam, 1998; Popham, 2000). Many assessment techniques, including 3


4

informal teacher-based intuitive judgments and formal assessments, can serve this purpose provided they inform teachers how to improve the quality and accuracy of instruction and/or enable students to improve their own learning. While recent assessment reform advocacy (Weeden, Winter, & Broadfoot, 2002) has rejected formal, standardised tests as being summative assessments of learning, rather than formative assessment for learning, the improvement conception does not require exclusion of formal testing (Newton, 2007). The second conception centres on the assumption that assessment should hold students individually accountable for their learning. Practices aligning with this conception include: assigning grades or scores to student work, judging student performance against criteria, and awarding certificates or qualifications based on performance. These data may be used to make decisions regarding retention, graduation, and tracking or streaming (Guthrie, 2002). The third conception posits that assessment should be used to evaluate a teacher's, a school's, or a system's use of society's resources (Firestone, Schorr, & Monfils, 2004). Here, assessment publicly demonstrates teacher and/or school effectiveness (Hamilton, 2003); at times these data are used to justify sanctions imposed on those not reaching required standards (Guthrie, 2002). The premise of the final conception is that assessment, usually defined as formal evaluation, has no legitimate place within teaching and learning; teachers’ intuitive judgments are all that are deemed necessary. Assessment is also rejected because of perceptions that it negatively affects teacher autonomy and professionalism and distracts attention away from student learning (Dixon, 1999). It may also be deemed unfair for certain students, causing unnecessary anxiety and damaging student selfesteem (Harris, Irving, & Peterson, 2008).

4


5

During the last decade, New Zealand has encouraged teachers to utilise more formative assessment practices, adopting the ‘assessment for learning’ rubric (Black et. al, 2003; Clarke, Timperley, & Hattie, 2003) explicitly aligned with the improvement conception. Formal qualifications are only issued during the final three years of New Zealand schooling; however, school and student accountability measures are still common. Even primary school students may be evaluated against curriculum objectives and leveled criteria, required to complete standardised assessment, or placed in leveled learning group within a class. This has created some angst for New Zealand teachers (Dixon, 1999) as more formal, standardised assessments or tests, which teachers frequently align with accountability and/or irrelevance conceptions, are often negatively contrasted with formative practices. Using the Teachers’ Conceptions of Assessment (TCoA) survey instrument, Brown and colleagues have investigated New Zealand , Australian (Brown, 2008), and Hong Kong (Brown, Kennedy, Fok, Chan, & Yu, in press) teachers’ attitudes towards improvement, student accountability, school accountability, and irrelevance conceptions. Across countries and sectors, teachers consistently had positive agreement means for both improvement and student accountability conceptions and low means for school accountability and irrelevance conceptions. By inspecting intercorrelations among the four factors, it appeared Australian and New Zealand teachers conceived student accountability as being irrelevant (i.e., as scores on student accountability increased, so did the values for irrelevant), while school accountability was positively correlated with improvement. In contrast, the Hong Kong teachers had strong positive correlations between improvement and student accountability. There were moderate differences between primary and secondary teachers in New Zealand and Queensland especially around the importance of student accountability. Together

5


6

these studies support the claim that teacher conceptions are likely to be influenced by their sector and school system. Study Design While the TCoA inventory research provides robust data relating to teachers’ conceptions of assessment, it has limitations. First, it measure responses to just four conceptions of the purpose of assessment (i.e., improvement, student accountability, school accountability, and irrelevance). While validating evidence was found during the development of the inventory (Brown, 2008), there may be significant purposes it does not capture. Additionally, previous administrations of the survey have been anonymous and have not allowed researchers to follow-up with participants about their reasons for responding as they did. The study reported in this paper adopted a qualitative approach to investigate New Zealand Years 5-10 teachers’ conceptions of assessment. The aim of the study was to ascertain whether conceptions other than those identified in the TCoA inventory existed. Phenomenography (Marton, 1981, 1986) was selected as the research approach as it is used to identify the range of conceptions present within a population. When conducting phenomenographic research, a second order perspective is adopted, orientating towards the participants’ points of view, forcing the research to set aside pre-existing assumptions based on personal experience and knowledge of the research literature. One strength of the phenomenographic approach is that it is based on the assumption that people hold multiple and, at times, contradictory conceptions within their frame of reference, making it impossible to claim that any particular participant ‘holds’ just one specific conception. However, these diverse conceptions can be aggregated to form an outcome space representing the variation of conceptions

6


7

present within the population. While phenomenography cannot be used to categorise individual participants into particular conception groups or quantify the number adopting a particular way of thinking, it is useful for exposing the range of variation within the sample and identifying logical relationships between these categories. This makes it suitable for identifying what, if any, conceptions of the purpose of assessment fall outside of Brown’s (2008) typology. Method Thirty six schools in the Auckland region participated in this study, part of the Measuring Teachers’ Assessment Practices (MTAP) project at The University of Auckland. These included primary (Years 1-6), full primary (Years 1-8), intermediate (Years 7 and 8), and high schools (Years 9-13) from across the school deciles (i.e., an index of socio-economic status with 1 being the lowest and 10 being highest). Teachers of Year 5 to 10 mathematics and/or English were invited to complete Brown’s TCoA-III questionnaire (Brown, 2006). An example of items from the questionnaire is shown in Table 1. In all, 161 teachers completed the questionnaire with over 100 agreeing to be interviewed. After the second author analysed the questionnaire results, he selected 26 teachers with noticeably different conception profiles for interview (Brown & Harris, 2008 provides details of questionnaire results). [Insert Table 1 around here] The first author then conducted semi-structured interviews and all subsequent analysis without awareness of participants’ questionnaire response profile. All data were transcribed verbatim and each utterance was labelled using Lankshear and Knobel’s (2004) method. For example, in the label L1:032, L referred to the

7


8

participant pseudonym (Lisa), 1 represented the first interview, and 32 indicated the 32nd utterance within the interview sequence. The first step in analysis was bracketing or excluding preconceived ideas (Ashworth & Lucas, 2000; Marton, 1994) to minimise researcher subjectivity. All codes used when classifying data were developed from the transcripts using participant words. Data were systematically and iteratively compared and contrasted with other participant data; at no point were pre-existing categories utilised. Analysis followed Marton’s (1986) procedures involving two main processes: creating categories of description and ordering the outcome space. Categories of description were formed by grouping together similar understandings, each representing a qualitatively different conception. After several readings of the data, utterances related to the research question were selected and marked using three indicators: 1. Frequency - how often an idea was articulated 2. Position - where the statement was situated; often the most significant elements were found in an answer’s beginning 3. Pregnancy - when participants explicitly emphasised that certain aspects were more important than others. (Sjostrom & Dahlgren, 2002) Once data relating to conceptions were identified, passages were analysed and interpreted within their contexts before being removed to create pools of meaning. Data were compared and contrasted, leading to movement between pools. Borderline cases were examined and criteria for each pool made explicit. Once the system was stable, data in the pools were abstracted into categories of description (Marton, 1981). The categories of description were then organised into an outcome space representing the ways participants understood the phenomenon. Bracketing no longer occurred; categories were compared with each other and with academic literature to

8


9

create a ‘logical’ model. Relationships between categories were primarily defined by similarities and differences (Marton & Saljo, 1997) and categories were ordered by complexity (Marton, 1994). The categories and final outcome space were thoroughly interrogated by the second author and found to be descriptive of the data. Results This study found that within this group of teachers, multifaceted conceptions of the purposes of assessment could be detected. The teachers reported assessing student achievement and behaviour formally and informally using a wide range of practices, with many utilising different types of assessment for differing purposes. Seven major purposes of assessment were abstracted from the interview data. These were (#1) compliance, (#2) external reporting, (#3) reporting to parents, (#4) extrinsically motivating students, (#5) facilitating group instruction, (#6) teacher use for individualising learning, and (#7) joint teacher and student use for individualising learning. Each is illustrated with a typical quotation in Table 2. [insert Table 2 about here] Teachers’ affective responses towards each category were one factor used to organise the outcome space (Figure 1). Responses ranged from negative to positive, with Category #1 being clearly negative, Categories #2 and #3 somewhat negative, and Category #4 neutral, while Categories #5 to #7 were positively evaluated. While all teachers discussed the first five categories, teachers were divided over whether assessment was for teacher use or joint teacher-student use, causing the model to branch at the top. Details of each category are discussed further below. [insert figure 1 about here] Category #1- Compliance

9


10

Within the compliance conception, the assessments discussed were said to be used primarily to fulfil Ministry mandates or school-wide directives. Standardised testing was talked about most frequently, but school based practices clashing with teachers’ personal beliefs about effective assessment were also mentioned. These practices were rejected as irrelevant, inaccurate, or negative for teachers, pupils, and learning. Many negative attitudes towards particular assessments came when teachers did not understand how the assessments would improve teaching and learning. As a high school teacher explained: Teachers get annoyed about it (asTTle) because it’s extra work, takes them out of teaching time, and you don’t see where it’s going, but the school is really pushing it. What we’re doing with it yet, I’m not too sure. (F1:032) Representative of standardised assessments in this category, the asTTle assessment was deemed irrelevant because it used class time without teachers seeing visible educational benefits. Some standardised assessments were viewed as inaccurate and unreliable, despite the quality assurance processes involved in developing such tasks. For example, a primary teacher said: Like the PAT [Progressive Achievement Test] test, the questions are quite wordy and their [students’] English is not so great, so they might know the answer, but the question confuses them and they get it all wrong. (L1:022) This concern was notably more expressed in the context of lower decile schools and where students were perceived as having low literacy or English language skills. Other factors, including the weather, cheating, students’ personal lives, and marking inconsistency were said to jeopardise assessment validity and reliability. Within this conception, caution towards standardised assessments was expressed because of the negative affective results they could have for students. For example, a primary teacher, describing mandated testing, said:

10


11

You see the looks on their faces, especially those ones that know they’re going to struggle… it almost broke my heart after she called herself dumb… a kid thinks of themself as dumb, you’re losing the battle already. So whenever you get a grade, it’s never great because a kid can see, “I only got 12 percent and everyone else got 60.” Who’s that really helping? (X1:166) As some students were perceived as ‘poor test takers,’ this mode of assessment was considered perpetually unfair for those pupils. In this category, certain assessments, often standardised ones, were said to be used simply to comply with mandates. These assessments were disliked and perceived as having little educational benefit for students. This conception also captured the tension teachers described between the requirement to report these kinds of evaluative data and their professional judgments that these results were inaccurate and/or would negatively categorise students. Category #2- External reporting While Category #1 (Compliance) evaluated standardised assessments negatively, the next category expressed agreement with the idea that schools should give evidence of achievement, conceding that reporting standardised assessment results was the most practical way of being accountable to external organisations (e.g. Education Ministry, school boards). Standardised data were used to measure the school’s academic success by comparing results with other schools and national benchmarks. This conception rejected the notion of holding individual students accountable for their scores, but was positive about its uses once school data were aggregated. However, within this category the concern was vocalised that heightened accountability requirements might cause schools to manipulate data and deliver narrow test-related instruction to students. In this category, some useful purposes were attributed to standardised assessment. For example, an intermediate teacher explained:

11


12

You want to know, particularly in a school like this, what achievement looks like in this school. And where do you target money? Where do you target teacher resources? Where do you target book resources? … Are the children actually progressing or are they standing still? Are we getting closer to norm? (U1:078) Decisions about funding and the school’s overall progress were said to be useful ways of utilising these data. Within this category, concern was raised that heightened accountability might lead to unethical manipulation of data. For example, a primary teacher stated: … There’s lies, damn lies, and then there’s statistics… if it becomes a situation where we are that accountable, schools will fib. They will tell lies or they will twist it … they will say to certain children, “you are not sitting this test because you’re going to bring our results down.” (M1:104) She later explained that if teachers and schools were judged by students’ results, they would ‘teach to the test’ and that this would be detrimental to student enjoyment of learning and the delivery of a well-rounded curriculum. Even under the current relatively low-stakes assessment regime, within this category concern was voiced about whole school data manipulation, especially at a secondary level. A tension also existed between external moderation and teacher professionalism. For example, a primary teacher explained: Because even without the assessment, the necessary assessment and reporting to ERO and so on, I still believe those things in the main would be done, in the vast majority of cases. (T1:104) He later explained that people became teachers for altruistic reasons and that standardised evaluation shifted attention away from children’s needs and devalued teacher professional judgments. Within this category, external reporting was described as a ‘necessary evil’ because standardised assessment was considered the most efficient way for schools to show accountability. However, concerns about data manipulation were frequently voiced, leading to significant negativity within this category.

12


13

Category #3- Reporting to parents While this third category also relates to reporting, within the data, parents and external organisations were described as different stakeholders with differing claims on assessment information. Reporting to parents was considered important, but concerns were raised about the kind of comparative data parents commonly requested and within a higher decile school context, parental pressure was cited as problematic. Within this category, parents were described as valuable stakeholders who could work with teachers to help students. Within a primary and intermediate context, a wider range of assessment types were said to be used for reporting than within secondary schools where “you’ve got to be able to back everything up” (V1:100). Despite an overall positive attitude towards reporting to parents, concerns were raised about the comparative data parents often requested. As an intermediate teacher explained: … what I’m trying to do is inform them [parents] of what their child can do and how we can work together to move the child on, but for parents … they want to know is my child average, above average, or below average. Sort of a bit of a clash there … the parent saying, “Is my kid average or above average or below average?” That’s about giving marks. And that’s about competition for the whole group. … (U1:078, 80) While the teacher-parent partnership was described as more tangible and productive for the students than external reporting, conflict existed between what was considered productive to report (e.g., data about the child’s abilities) and what parents wanted (e.g., marks). Quantitative comparisons between children were of concern as they were believed to create winners and losers. Within a higher decile school context, parental expectations of both teachers and students were highlighted as an area of concern. As a Decile 8 high school teacher explained:

13


14

Yeah, they’ll [parents] push the child, and the child will raise their standard or focus or whatever, but that doesn’t explain why teachers make sure they keep telling us [new teachers], “Keep a record of all the stuff that you’ve been doing and if you’ve handed out things to help them, make sure you’ve got a record of it because when the parents do come back at you, you can cover your ass.” … Obviously I’d better do that because there will be a time where parents will put it on me. (F1:124) Extensive record keeping was seen as one way of helping teachers defend their judgments and avoid blame for poor student performance. In this category, sharing student results with parents was considered necessary and important. However, concerns were raised over the kind of comparative data parents often wanted and the unrealistic expectations some had of teachers and students; these tensions led to some negativity within this category. Category #4- Extrinsically motivating students In addition to reporting, extrinsically motivating students was described as a major purpose of assessment. Students were described as motivated by both competition or pressure and enjoyment or praise. Competition and pressure were thought to motivate students because pupils feared failure and wanted success. Formal tests used for reporting (see Categories #2 and #3) were frequently mentioned in this context. Good grades and qualifications (e.g., NCEA credentials) were considered rewards for hard work, although poor grades were said to cause some to quit trying. For example, a high school teacher explained: A student wants to pass.… you don’t want to fail. So unless you can judge an informal assessment with a mark, they’re not really interested. Rewards system I guess. (V1:226) External qualifications like NCEA were also said to be student incentives. Within primary and intermediate contexts, there was division over whether tests and grades were positive motivators. Some shared the view of an intermediate teacher, who said:

14


15

I have found that they need a summative, a formal kind of test situation, contrary to what I have been told. For a while it was a swear word. “We don’t use tests and you’re not allowed to.” … There’s that part in every person that wants to feel that they’ve achieved something. That they’ve mastered it. (A1:044) However, others adopted the view expressed by the following primary teacher: … if you have straight assessment where peers, they all get a mark, it can be a negative influence on them… If it was a test and it was out of 10, then the ones who get 9 and 10 are the ones who are yahooing it and the ones who only get 4 or 5 are the ones who tend to think, “Oh, I’m not as good as they are.” (H1:100, 102) Here, tests and grades or scores were described as encouraging high achieving students but potentially discouraging lower achievers. Enjoyment and praise were also described as motivating within an assessment context. As a primary teacher explained: … praise is an excellent way to keep them going. So if I walk past and I say “Oh my gosh, look at that wonderful sentence” or “I love your impact beginning,” or something then they’re like, “Oh, I’ve got to write good,” and they keep writing. (L1:100) Praise and enjoyment were not seen as having a negative side like competition and pressure. Within this category, assessment was described as a way of motivating students. This could be either through competition and pressure, viewed as having potentially negative side effects for some students, or via enjoyment and praise, which was considered positive. However, both types of motivators discussed here were extrinsic; attitudes towards them were generally neutral, with few participants overtly positive or negative. Category #5- Facilitating group instruction Using group instruction to effectively facilitate teaching and learning was a fifth purpose of assessment. Ability grouping, through streamed classes or differentiated small group instruction, was described as best instructional practice.

15


16

When grouping pupils, standardised test data and teacher input were both used. As a high school teacher explained: When it comes to placing kids, initially it’s taken on their test results. Then, if there’s any students that teachers think are wrongly placed, regardless of their test results, then teacher recommendation comes in as well. (V1:086) Here judgments are blended with test data, contrasting with Categories #2 and #3 where test data were privileged for reporting. To maximise benefits from group instruction, it was said that students needed to be prepared for assessments they might encounter at high school or university. For example, an intermediate teacher explained: …we’re trying to prepare children … it’s all very well having this formative assessment and “gosh, you’re doing well here” … but we’ve got to be realistic that there’s an exam world out there and we’ve got to prepare them for exams and if they go to Grammar, day 1 they have exams, … And also university’s all about exams. (C1:072) Despite expressing dislike for these practices, it was said students would be disadvantaged if not exposed to these assessments. Included in this category were discussions about assessments that reinforced classroom routines, rules, and behavioural expectations needed for group instruction. For example, a high school teacher explained: I call them quick fire quizzes… it’s 5 random questions about the book… They should get all five, but if they can’t even get one, it shows me that they haven’t read the novel, although they are probably claiming to have read it. (E1:036) These assessments helped teachers manage group instruction by letting them know if work was completed, while potentially motivating students to complete tasks. Additionally, the presentation of work and students’ work habits were said to be assessed. In this category, assessment results were described as useful for streamlining group instruction. This was done through the grouping of students for instructional

16


17

purposes; additionally, students were thought to need preparation for future assessment and development of the skills needed to participate in classroom work. These were described positively as benefiting all students, not any particular students individually. Category #6- Teacher use for individualised teaching In category #6 assessment data was said to be used to plan future teaching, modify instruction, and judge when information should be retaught on a more individual basis. Correcting errors and giving feedback were considered important teacher roles; pupils were not expected to make sense of assessment results. Assessment within this category was described as a primarily teachercontrolled exercise. For example, a high school teacher explained: … if I’ve observed a particular student is struggling with the vocabulary of a unit, then for the next task or activity, I might give them less words to look up or provide more meanings…. (N1:050) Tools like pre-tests (used to ensure work was appropriate) and post-tests (used measure learning and map future instruction) were commonly cited. Assessment was also said to provide teachers with feedback about their effectiveness so they could improve teaching. While not all results were said to be shared with students, most were. For example, a full primary teacher explained: … once they’ve sat their tests and I’ve marked it and I give it back to them, we go over it. Some of them I do one to one conferencing, just to say, “Well listen, you just missed out because of this, and this is why. Why did you do that?” So I find out what they’re thinking and then hopefully I can steer them to the right track… (W1:030) It was said to be the teacher’s job to figure out what the student needed to do to improve.

17


18

Within this category, assessment results were said to be used to modify teaching for the benefit of individual students. Use of assessment data remained a teacher controlled exercise. Assessments that could be used to help teachers diagnose and fix individual student problems were described favourably. Category #7- Joint teacher and student use for individualised learning In this final category, joint teacher-student uses of assessment were foregrounded, instead of the teacher-controlled processes described in Category #6. Sharing ownership of the data and its interpretation with students through a range of primarily formative practices was considered best practice. Using assessment jointly with students was thought to help pupils become independent learners. For example, a primary teacher stated: … once they understand that [shared assessment criteria] and they start applying it, then they’ve got that, “Oh, I can see where I have to go next. I don’t have to have the teacher tell me.” And what we’ve just started doing now is like, “You tell me what you want me to do with the lesson? What are your weak areas that you don’t understand?” so they’ve just started doing that. (G1:050) Passages like this highlight practices where students use assessment data to determine what they needed to learn next. Self- and peer-assessment mechanisms were commonly reported within this category. To allow students control, some teachers like this primary teacher reported modifying assessment tasks and criteria: … they [students] need to be interested in how they’re being assessed .… they [students] do look at the surface features more, where teachers will look at the deeper features, but that’s okay because if that’s important to them, then it’s important to us in the end because they’re the stakeholders. (X1:072) Practices like co-marking work with the child and getting students to present assessment data to parents were put forward. For example, one primary teacher explained, “I mark it [their work] with them. They bring it to me and we talk about

18


19

where to next” (M1:022). This process was described as powerful because it helped students identify their own mistakes. Enacting these kinds of practices consistently was described as challenging because of existing school and community expectations. For example, one primary teacher explained: We use rubrics here a lot. … we sort of sit around a staff meeting and we sort of make it up ourselves, when really the kids should have buy-in. “What do you think someone that learns very well does when we’re studying this?” And so they should be able to say, “They do this; they do that.” (X1:054) Another challenge was the perception that parents wanted teacher evaluations of student learning, not pupil opinions. In this final category, assessment processes and results were said to be jointly constructed with students so pupils had input about the direction of future learning. While students were said to need considerable input, it was acknowledged that this was difficult in light of some school and community expectations. Student growth and improvement were described as the outputs of this way of conducting assessment, leading to an overtly positive attitude towards it. Discussion Within this data set, seven major purposes for assessment were described. These data show teachers’ conceptions of assessment are affected by multiple pressures including: protecting students from harm, fulfilling employer obligations, seeking to improve schools, and helping students learn. These obligations mean teachers must juggle multiple stakeholders’ interests when deciding how to assess students and use these data. The seven categories reported in this study generally correspond with three major purposes for assessment: irrelevance, accountability, and student improvement

19


20

and, hence, provide independent validation of the categories used in Brown’s TCoAIIIA inventory and support continued use of the inventory in research contexts. The first category, Compliance, included assessments teachers viewed as irrelevant, inaccurate, or negative but were required to administer. The next three categories (#2- External reporting, #3- Reporting to parents, #4- Extrinsically motivating students) were all related to accountability. While the first two categories were primarily about school accountability through the reporting of student results, Category #4 contained elements of the student accountability construct when discussing the role of grades and external qualifications on student motivation. However, within the data set, pressure and competition were not considered to positively motivate all students as lower achieving students might suffer negative affective consequences through such evaluations. Therefore, it appeared that some teachers within this sample disagreed that student accountability was a valid purpose for assessment. While these data contrasted with Brown’s (2008) findings that New Zealand teachers strongly agreed with student accountability questionnaire prompts, they were consistent with his finding that student accountability was correlated with irrelevance. The final three categories (#5- Facilitating group instruction, #6- Teacher use for individualising learning, #7- Joint teacher and student use for individualising learning) all showed an improvement orientation towards assessment. Category #5 differed from the last two as it focused on how assessment could benefit the group rather than the individual learner. Categories #6 and #7 divided teachers, with some proposing assessment was a teacher tool while others indicated it should be used jointly with students. Category #7 is viewed as most aligned with the child-centred pedagogy promoted in New Zealand (Fraser, 2001; Vaughan &Weeds, 1994).

20


21

Nevertheless, Category #6 has been argued as an essential characteristic of school-based assessment for improvement (Hattie & Brown, 2008; Parr & Timperley, 2008).

While there are many obvious tensions between these assessment purposes, teachers reported most difficulty with balancing school and student needs. For example, some practices like standardised testing were seen as potentially useful for schools as they provided a holistic picture of achievement, but as negative and often inaccurate when examining individual student scores. Brown’s (2008) survey research found that the school accountability conception was positively correlated with improvement, which would make sense if teachers saw school accountability as potentially improving their school’s practices. To better illustrate the tensions teachers reported between the interests of students and schools, Figure 2 plots major constructs teachers discussed within the categories on two axes. The x axis represents a continuum of positive and negative value for schools, while the y axis represents positive and negative value for students. The field of the figure is further divided by two arcs which separate the three purposeoriented conceptions of assessment. The bottom arc represents irrelevance, the top arc is improvement and the middle space is accountability. [insert figure 2 about here] Assessments that teachers reported ignoring or rejecting were evaluated by them as being negative for both students and schools. The middle region captured the accountability-oriented categories around which teachers were slightly positive or negative. Finally, the improvement field captures categories that were deemed as positive for students and for schools as well. The most tension occurred with accountability practices, seen as necessary for schools, but as sometimes having negative consequences for students. Also, the placement of Categories #6 and #7

21


22

highlights another tension. While including students in the process of interpreting and acting on assessment was seen as extremely positive for student (Category #7), some quotations (e.g. X1:072 which highlighted that students often place importance on surface rather than deep features) demonstrated that students do not necessarily direct curriculum and assessment in the ways schools required. Category #6 was less positive for students than Category #7, but had fewer tensions with what was beneficial for schools as teacher control made for easier accommodation of curriculum and policy. This study suggested that teachers aligned with Categories #6 and #7 practice assessment quite differently despite working towards the same purpose: student improvement. More empirical evidence is needed to show whether the joint studentteacher practices (Category #7) lead to more improvement than teacher directed practices (Category #6). For example, New Zealand secondary students reported they find self and peer assessment practices, often seen as central to the assessment for learning approach (e.g., Black & Wiliam, 1998), of little use; only teacher feedback was considered accurate and useful for improvement purposes (Brown, Irving, Peterson, & Hirschfeld, 2009; Peterson & Irving, 2008). While there may need to be more research into how student-directed assessment strategies could be effectively implemented in schools, teachers must also adopt approaches based on Categories #5 and #6. Students rightly depend on teachers for instruction and feedback. In order to involve students in assessment interpretation and design, it seems necessary that teachers are actively doing this themselves as well. This study has shown that teachers see assessment as having a range of diverse purposes aligned with different practices. These data highlighted key tensions in teachers’ thinking about assessment—school vs. student; compliance vs.

22

Comment [O1]: Has this been published yet??


23

improvement; positive vs. negative evaluation. These tensions need to be resolved within each teacher and school so that educational assessment can fulfil its real purpose of improving outcomes (Popham, 2000).

23


24

References Ajzen, I. (2005). Attitudes, personality and behavior (2nd ed.). New York: Open University Press. Ashworth, P., & Lucas, U. (2000). Achieving empathy and engagement: A practical approach to the design, conduct and reporting of phenomenographic research. Studies in Higher Education, 25(3), 295-308. Black, P., Harrison, C., Lee, C., Marshall, B., & Wiliam, D. (2003). Assessment for learning: Putting it into practice. Maidenhead, UK: Open University Press. Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education, 5(1), 7-74. Brown, G. T. L. (2006). Teachers' conceptions of assessment: Validation of an abridged instrument. Psychological Reports, 99, 166-170. Brown, G. T. L. (2008). Conceptions of assessment: Understanding what assessment means to teachers and students. New York: Nova Science Publishers. Brown, G. T. L., & Harris, L. (2008, November). Teachers' conceptions of assessment: Interview validation of survey responses. Paper presented to the New Zealand Association for Research in Education (NZARE) annual conference, Palmerston North, NZ. Brown, G. T. L., Irving, S. E., Peterson, E. R., & Hirschfeld, G. H. F. (2009). Use of interactive-informal assessment practices: New Zealand secondary students’ conceptions of assessment. Learning & Instruction,19(2), 97-111. Brown, G. T. L., Kennedy, K. J., Fok, P. K., Chan, J. K. S., & Yu, W. M. (in press). Assessment for improvement: Understanding Hong Kong teachers’ conceptions and practices of assessment. Assessment in Education: Principles, Policy and Practice.

24


25

Brown, G. T. L., & Lake, R. (2006, November). Queensland teachers' conceptions of teaching, learning, curriculum and assessment: Comparisons with New Zealand teachers. Paper presented at the Annual Conference of the Australian Association for Research in Education (AARE), Adelaide, Australia. Clark, C., & Peterson, P. (1986). Teachers' thought processes. In M. Wittrock (Ed.), Handbook of research on teaching. (3rd ed., pp. 255-296). New York: MacMillan. Clarke, S., Timperley, H. S., & Hattie, J. A. (2003). Unlocking formative assessment: Practical strategies for enhancing students' learning in the primary and intermediate classroom (New Zealand Ed.). Auckland, NZ: Hodder Moa Beckett. Dixon, H. (1999). The Effect of Policy on Practice: An Analysis of Teachers' Perceptions of School Based Assessment Practice. Unpublished Masters Thesis, Massey University, Albany, NZ. Firestone, W. A., Schorr, R. Y., & Monfils, L. F. (Eds.). (2004). The ambiguity of teaching to the test: Standards, assessment, and educational reform. Mahwah, New Jersey: Lawrence Erlbaum Associates. Fraser, D. (2001). Developing classroom culture: Setting the climate for learning. In C. McGee & D. Fraser (Eds.), The Professional Practice of Teaching (2nd ed., pp. 15-34). Palmerston North, NZ: Dunmore Press. Guthrie, J. T. (2002). Preparing students for high-stakes test taking in reading. In A. E. Farstrup & S. J. Samuels (Eds.), What research has to say about reading instruction (pp. 370-391). Newark, DE: International Reading Association. Hamilton, L. (2003). Assessment as a policy tool. Review of Research in Education, 27, 25-68.

25


26

Harris, L. R., Irving, S. E., & Petersen, E. (2008, December). Secondary teachers' conceptions of the purpose of assessment and feedback. Paper presented to the Annual Conference of the Australian Association for Research in Education (AARE), Brisbane, Aus. Hattie, J. A., & Brown, G. T. L. (2008, September). Tensions in educational assessment and measurement in New Zealand: A national reporting engine and conceptions of assessment. Paper presented at the 1st Educational Psychology Forum, Auckland, NZ. Heaton, J. B. (1975). Writing English language tests. London: Longman. Lankshear, C., & Knobel, M. (2004). A handbook for teacher research. Berkshire: Open University Press. Marton, F. (1981). Phenomenography- Describing conceptions of the world around us. Instructional Science, 10, 177-200. Marton, F. (1986). Phenomenography- A research approach to investigating different understandings of reality. Journal of Thought, 21(3), 28-49. Marton, F. (1994). Phenomenography. In T. Husen & T. N. Postlethwaite (Eds.), The International Encyclopedia of Education (Vol. 8, pp. 4424-4429). New York: Pergamon. Marton, F., & Saljo, R. (1997). Approaches to learning. In F. Marton, D. Hounsell & N. Entwistle (Eds.), The experience of learning (pp. 39-58). Edinburgh: Scottish Academic Press. Muijs, D., & Reynolds, D. (2005). Effective teaching: Evidence and practice. London: Sage Publications. Newton, P. E. (2007). Clarifying the purposes of educational assessment. Assessment in Education: Principles, Policy & Practice, 14(2), 149-170.

26


27

Parr, J. M., & Timperley, H. (2008). Teachers, schools and using evidence: Considerations of preparedness. Assessment in Education: Policy, Principles and Practice, 15(1), 57-71. Peterson, E. R., & Irving, S. E. (2008). Secondary school students' conceptions of assessment and feedback. Learning and Instruction, 18, 238-250. Popham, W. J. (2000). Modern educational measurement: Practical guidelines for educational leaders (6th ed.). Boston: Allyn & Bacon Shohamy, E. (2001). The power of tests: A critical perspective on the uses of language tests. Harlow, UK: Pearson Education. Sjostrom, B., & Dahlgren, L. O. (2002). Nursing theory and concept development or analysis: Applying phenomenography in nursing research. Journal of Advanced Nursing, 40(3), 339-345. Thompson, A. G. (1992). Teachers' beliefs and conceptions: A synthesis of the research. In D. A. Grouws (Ed.), Handbook of research on mathematics teaching and learning (pp. 127-146). New York: MacMillan. Torrance, H., & Pryor, J. (1998). Investigating formative assessment: Teaching, learning and assessment in the classroom. Buckingham, UK: Open University Press. Vaughan, L., & Weeds, A. (1994). Managing an effective classroom. In C. McGee & D. Fraser (Eds.), The Professional Practice of Teaching: An Introduction to Teaching, Learning and Curriculum (pp. 131-174). Palmerston North, NZ: Dunmore Press. Warren, E., & Nisbet, S. (1999). The relationship between the purported use of assessment techniques and beliefs about the uses of assessment. In J. M. Truran & K. M. Truran (Eds.), 22nd Annual Conference of the Mathematics

27


28

Education and Research Group of Australasia (Vol. 22, pp. 515-521). Adelaide, SA: MERGA. Webb, N. L. (1992). Assessment of students' knowledge of mathematics: Steps toward a theory. In D. A. Grouws (Ed.), Handbook of research on mathematics teaching and learning (pp. 661-683). New York: Macmillan. Weeden, P., Winter, J., & Broadfoot, P. (2002). Assessment: What's in it for schools? London: RoutledgeFalmer.

28


29 Comment [HKIEd2]: I am not happy. These items have all been published elsewhere—see Brown 2006 Psychological Reports—and we did not get permission to reprint. So we must not do this. Also all items were printed in teh 2004 AiE paper in a diagram. So i do not accept that it is legitimate to reprint so much of the questionnaire. 1 item per category could be fair dealing but it should be referenced at least in the note under the table. also your version did not show conception.

Table 1- Example of prompts from TCoA-III inventory Conceptions of Assessment Assessment is for School Accountability Assessment provides information on how well schools are doing Assessment is for Student Accountability Assessment places students into categories Assessment is for Improvement Assessment is a way to determine how much students have learned from teaching Assessment is Irrelevant Assessment forces teachers to teach in a way against their beliefs Note. All items previously disclosed in Brown (2004, 2006).

29


30

Table 2- Representative quotes for each category of description Category

Quotation

#1 Compliance

… One of the purposes is to conform with what is required under the legislation… that is why I think a lot of it is done within schools. I’m not just talking about this school. (T1:086)

#2 External

I think it’s important that schools are answerable to what is

reporting

happening. And if everything’s standardised, then you can say, “You’re performing well; you’re not. Why not?” And if you can answer that, there’s no problem. If you can’t answer that, then there is a problem. (V1:190)

#3 Reporting to

I think that’s obvious. There needs to be some sort of reporting

parents

because parents want to know and that’s important to them and parents need that buy-in. (X1:094)

#4 Extrinsically

“If I don’t get excellence on this test, my mum’s going to dah, dah,

motivating

dah,” so good motivational tools, assessments. (F1:176)

students #5 Facilitating

I’m not going to choose it [groupings] based on the colour of each

group instruction

child’s hair, so I have to do assessment. So it’s for [the] practical implication of making a classroom program effective and manageable. (A1:068)

#6 Teacher use

… It’s us [teachers] looking at the mark and thinking, “This child is

for

way below; there’s things we need to do to boost this child.” … it’s

individualising

a teacher’s tool and the way we choose to use it will impact on our

learning

student’s development. I think it’s for us, and I don’t believe it’s a student tool. (R1:066)

Table continued 30


31

Category

Quotation

#7 Joint teacher

I’d like to think of it [assessment] as more of a joint thing…. It has

and student use

to be seen by the children and by me as a joint effort. We’re going to

for

find out how much we’ve actually learned… [and] where do we

individualising

have to go next... (M1:060)

learning

31


32

Authors’ notes

Lois Harris is a Research Post-doctoral Fellow at University of Auckland. Her research examines relationships between educational stakeholders’ thinking and their practices, with recent studies investigating assessment and student engagement. She also has a strong interest in both qualitative and quantitative research methodology. Previously, she has been a secondary school teacher in the United States and Australia.

Dr. Brown is an Associate Professor in the Department of Curriculum and Instruction at the Hong Kong Institute of Education. His research interests are assessment and evaluation and the effects of test-taker and test-user attitudes towards assessment on practices and outcomes. He developed and won funding for the Measuring Teachers’ Assessment Practices project while he was a Senior Lecturer at the University of Auckland, Faculty of Education.

32


33

Figure Captions Figure 1- Outcome space: Teacher conceptions of the purpose of assessment Figure 2- Diagram showing school/student tensions within teachers’ conceptions of assessment

33


Positive

A f f e c t i v e R e s p o n s e

Category # 6 Teacher use for individualising learning

34

Category #7 Joint teacher and student use for individualising learning

Category #5 Facilitating group instruction

Neutral

Category #4 Extrinsically motivating students

Category #2 External reporting

Category #3 Reporting to parents

Category #1 Compliance Negative

34


35

35