Students' Conceptions of Assessment

2017

Students’ Conceptions of Assessment: New Zealand

CODEBOOK/DATA DICTIONARY GAVIN T.L BROWN

THE UNIVERSITY OF AUCKLAND | FACULTY OF EDUCATION AND SOCIAL WORK

Table of Contents 1.

Purpose of this codebook ................................................................................................. 3

2.

About Students’ Conceptions of Assessment................................................................. 4

References ............................................................................................................................................... 4 Overview ............................................................................................................................................. 4 SCoA v1 .............................................................................................................................................. 4 SCoA v2 .............................................................................................................................................. 5 SCoA v3 .............................................................................................................................................. 5 SCoA v4 .............................................................................................................................................. 5 SCoA v5 .............................................................................................................................................. 5 SCoA v6 .............................................................................................................................................. 5 Multi-Site Invariance Studies ............................................................................................................. 5 Coverage ................................................................................................................................................. 6 Methodology ........................................................................................................................................... 6 Response Format ................................................................................................................................ 6 Missing data analysis ......................................................................................................................... 6 Privacy, security, or confidentiality issues ......................................................................................... 7 Coding Scheme ....................................................................................................................................... 7

3.

List of Data Sets ............................................................................................................... 8

4.

Data Source Variables ................................................................................................... 11

5. 6. 7. 8. 9.

Summary Table ................................................................................................................................. 11

School Measures ............................................................................................................. 12

Summary Table ................................................................................................................................. 12

Participant Demographic Measures ............................................................................. 16

Summary Table ................................................................................................................................. 16

Assessment Practice Checklist ...................................................................................... 18

Summary Table ................................................................................................................................. 18

Students’ Conception of Assessment Version Six and Five ....................................... 21

Summary Table ................................................................................................................................. 21

Students’ Conception of Assessment Version Five Additional Items ....................... 28

Summary Table ................................................................................................................................. 28

10.

Students’ Conception of Assessment Version Four ................................................ 31

11.

Students’ Conception of Assessment Version Three .............................................. 41

12.

Students’ Conception of Assessment Version Two ................................................. 46

13.

Students’ Conception of Assessment Version One ................................................. 51

Summary Table ................................................................................................................................. 31 Summary Table ................................................................................................................................. 41 Summary Table ................................................................................................................................. 46 Summary Table ................................................................................................................................. 51

1. Purpose of this codebook This codebook or data dictionary provides users of the Students’ Conceptions of Assessment Inventory details of the structure of the data files used to analyse the New Zealand data. This codebook needs to be read in conjunction with the following complementary files: 1. The Students’ Conceptions of Assessment Questionnaire or Inventory a. Version 6 b. Version 5 c. Version 4 d. Version 3 e. Version 2 i. Form 1 ii. Form 2 f. Version 1 i. Form A ii. Form B iii. Form C iv. Form D 2. The New Zealand data files, in SPSS format a. Version 6 b. Version 5 c. Version 4 d. Version 3 e. Version 2 f. Version 1 3. The confirmatory factor analysis input files for all versions a. AMOS graphic input system

2. About Students’ Conceptions of Assessment The Students’ Conceptions of Assessment (SCoA) inventory is a self-administered, self-report opinion, attitude, or belief questionnaire designed in New Zealand by Gavin T L Brown (2003-2008) after his PhD dissertation research into Teacher Conceptions of Assessment. That work was carried out with New Zealand secondary school students. It started on the assumption that student conceptions of assessment may be parallel to teacher conceptions. Very quickly it morphed into a model, now informed by self-regulation of learning theories about how students can improve their performance by using assessment to regulate their own learning practices. The questionnaire has been published in six gradually improved and developed versions. See list below for publications related to each version. Comparison studies between New Zealand data and responses of students in other jurisdictions has also been conducted and published. The most recent version of the inventory aggregates student beliefs about assessment around four major ideas: 1. Assessment is used by students to improve their learning and by teachers to guide students into improved learning. [Improvement] 2. Assessment is linked to external attributes of the student such as their future performance or job, their intelligence, and the quality of the school they attend. [External Attribution] 3. Assessment requires and builds on the quality of student cooperation and classroom climate and elicits strong positive emotion of enjoyment [Affect] 4. Assessment is oppressive, inaccurate, and ignored by the student [Irrelevance] In Version 5, these four ideas were created by 8 1st order factors which grouped in pairs into the four factors. In Version 6, one of the contributing 1st-order factors within Irrelevance had to be removed with the items loading directly on the main factor. The four major factors are inter-correlated. This means that the inventory is multi-dimensional and there is no single overall score. Reading of the listed references will provide the user with a sufficient background to appreciate the meaning and intent of the TCoA.

References Overview Brown, G. T. L. (2011). Self-regulation of assessment beliefs and attitudes: A review of the Students’ Conceptions of Assessment inventory. Educational Psychology, 31(6), 731-748. doi:10.1080/01443410.2011.599836 SCoA v1 Brown, G. T., & Hirschfield, G. H. (2007). Students' Conceptions of Assessment and Mathematics: Self-Regulation Raises Achievement. Australian Journal of Educational & Developmental Psychology, 7, 63-74.

SCoA v2 Brown, G. T. L., & Hirschfeld, G. H. F. (2008). Students’ conceptions of assessment: Links to outcomes. Assessment in Education: Principles, Policy and Practice, 15(1), 3-17. doi:10.1080/09695940701876003 Hirschfeld, G. H. F., & Brown, G. T. L. (2009). Students’ conceptions of assessment: Factorial and structural invariance of the SCoA across sex, age, and ethnicity. European Journal of Psychological Assessment, 25(1), 30-38. doi:10.1027/10155759.25.1.30 Walton, K. F. (2009). Secondary students’ conceptions of assessment mediated by self-motivational attitudes: Effects on academic performance. (Unpublished M.Ed. thesis), University of Auckland, Auckland, NZ. SCoA v3 Brown, G. T. L. & Hirschfeld, G. H. F. (2005). Secondary school students’ conceptions of assessment. Conceptions of Assessment and Feedback Project Report #4. Auckland: University of Auckland. SCoA v4 Brown, G. T. L. (2006, September). Secondary school students’ conceptions of assessment: A Survey of Four Schools. Conceptions of Assessment and Feedback Project Report #5. Auckland, NZ: University of Auckland. SCoA v5 Brown, G. T., Irving, S. E., Peterson, E. R., & Hirschfeld, G. H. (2009). Use of interactive–informal assessment practices: New Zealand secondary students' conceptions of assessment. Learning and Instruction, 19(2), 97-111. doi:10.1016/j.learninstruc.2008.02.003 Weekers, A. M., Brown, G. T. L., & Veldkamp, B. P. (2009). Analyzing the dimensionality of the Students' Conceptions of Assessment (SCoA) inventory. In D. M. McInerney, G. T. L. Brown & G. A. D. Liem (Eds.), Student perspectives on assessment: What students can tell us about assessment for learning. (pp. 133157). Charlotte, NC US: Information Age Publishing. SCoA v6 Brown, G. T. L., Peterson, E. R., & Irving, S. E. (2009). Beliefs that make a difference: Adaptive and maladaptive self-regulation in students’ conceptions of assessment. In D. M. McInerney, G. T. L. Brown, & G. A. D. Liem (Eds.), Student Perspectives on Assessment: What Students can Tell us about Assessment for Learning (pp. 159-186). Charlotte, NC: Information Age Publishing. AlQassab, M. (2012). Differential item functioning according to subject motivation: Psychometric properties of the Student Conceptions of Assessment (unpublished master's thesis), The University of Auckland, Auckland, NZ. Multi-Site Invariance Studies Brown, G. T. L. (2013). Student conceptions of assessment across cultural and contextual differences: University student perspectives of assessment from Brazil, China, Hong Kong, and New Zealand. In G.A.D. Liem & A. B. I. Bernardo (Eds.), Advancing Cross-cultural Perspectives on Educational Psychology: A

Festschrift for Dennis McInerney (pp. 143-167). Charlotte, NC: Information Age Publishing. Matos, D. A. S., Cirino, S. D., Brown, G. T. L., & Leite, W. L. (2013). A avaliação no ensino superior: Concepções múltiplas de estudantes Brasileiros [Assessment in higher education: Multiple conceptions of Brazilian students]. Estudos em Avaliação Educacional, 24(54), 172-193. doi:10.18222/eae245420131907 · Matos, D. A. S., & Brown, G. T. L. (2015). Comparing university student conceptions of assessment: Brazilian and New Zealand beliefs. In C. Carvalho & J. Conboy (Eds.). Feedback, Identidade, Trajetórias Escolares: Dinâmicas e Consequências (pp. 177-194). Lisbon, Portugal: Universidade de Lisboa, Instituto de Educação.

Coverage This manual applies to the New Zealand secondary student data files only and analysis files obtained from surveys carried out between 2003 and 2009 as described in the reference list. Only the SCoA items and accompanying checklist of assessment practices (used in Versions 4-6 only) are included.

Methodology The SCoA inventory involves two parts each with its own response format. The SCoA main inventory (Part A) consists of 33 items which probe student agreement about the four main purposes of assessment. Part B is a checklist which allows students to identify the types of practices they were thinking about when answering the main part. Response Format Each statement of the SCoA required the participant to respond on a six-point positively-packed rating scale. Options were scored: • Strongly Disagree=1; • Mostly Disagree=2; • Slightly Agree=3; • Moderately Agree=4; • Mostly Agree=5; and • Strongly Agree=6. The Assessment Practices Checklist required a binary Yes/No response. Participants were allowed to indicate as many as they considered relevant. Missing data analysis Prior to imputation, all cases with more than 10% missing were removed from further analysis. The remaining cases and variables had the remainder of missing values imputed using the expectation maximisation procedure. All imputed values 6.00 were corrected to either 1.00 or 6.00 respectively. All values between 1.00 and 6.00 were left uncorrected on the assumption that the response scale created a continuous variable.

Privacy, security, or confidentiality issues All data files have been anonymized so that individuals and their schools cannot be identified. All data were obtained under various institutional research board authorisations at the University of Auckland.

Coding Scheme Throughout this document, the data sets, and the excel master spreadsheet, SCoA statements have been coded to help align data across all six versions of SCoA. Statements have been assigned a two or three letter code to capture the content focus of the item plus a number. The lettering refers to the variable group or variable subgroup a statement belongs to. Lower numbers have been assigned to newer version of SCoA (6 and 5) and higher numbers are most likely aligned with earlier versions of SCoA. Where a statement has been repeated in different versions the coding is exactly the same. This also applies for statements where the wording was slightly different but which was intended to measure the same content. This is mainly seen in Version 1, which was the pilot version. Please see that section for further details. The content code structure is: Code Code Translation bd Bad ce Class environment ig Ignore ir Irrelevant pe Personal enjoyment sf Student future si Student improvement sq School quality sr Self-regulate sta Student accountability ti Teacher improvement val Valid

3. List of Data Sets The following data files capture the New Zealand data from the Students’ Conceptions of Assessment questionnaires carried out and collected from 2003 to 2009.

SCoA Master: The following Excel spreadsheet contains all items from all six versions and shows the history of items as they were added, deleted, or carried through. It also shows the factor to which each items was aligned. o NZ Master SCoA.xlsx

SCoA-VI Data Sets: The following SPSS data file contains data information that was collected in 2007 from large samples of students in a few Auckland region schools for Version 6 of SCoA. Note that Version 6 is a reanalysis of the data collected for Version 5. This SPSS file contains all relevant variables and codes as outlined within this codebook/data dictionary. o NZ SCoA-VI.sav The following AMOS file is the confirmatory factor analysis input for the SCoA-VI data set. o NZ SCoA-VI.amw

SCoA-V Data Sets: The SPSS data file contains data information that was collected in a national survey in 2006 of NZ secondary students, primarily in Years 9 and 10, for Version 5 of SCoA. This SPSS file contains all relevant variables and codes as outlined within this codebook/data dictionary. o NZ SCoA-V.sav The following AMOS file is the confirmatory factor analysis input for the SCoA-V data set. o NZ SCoA-V.amw

SCoA-IV Data Sets: The SPSS data file contains data information that was collected in 2006 from a survey of students in 4 Auckland regional secondary schools participating in the Conceptions

of Assessment and Feedback Project. This SPSS file contains all relevant variables and codes as outlined within this codebook/data dictionary. o NZ SCoA-IV.sav The following AMOS file is the confirmatory factor analysis input for the SCoA-IV data set. o NZ SCoA-IV.amw

SCoA-III Data Sets: The SPSS data file contains data information that was collected in 2005 from NZ secondary students in 4 Auckland regional school participating in the Conceptions of Assessment and Feedback Project. This SPSS file contains all relevant variables and codes as outlined within this codebook/data dictionary. o NZ SCoA-III.sav The following AMOS file is the confirmatory factor analysis input for the SCoA-III data set. o NZ SCoA-III.amw

SCoA-II Data Sets: The SPSS data file contains data information that was collected in 2004 from a national survey conducted to evaluate new reading comprehension test items as part of the Assessment Tools for Teaching and Learning Project. Items were split into 2 forms to be completed by the students after they had finished the trial tests. Only 11 items were common between the two forms. Thus analysis by Brown & Hirschfeld (2008) and Hirschfeld and Brown (2009) applies to those common items. Walton (2009) provides an overview of all the items. This SPSS file contains all relevant variables and codes as outlined within this codebook/data dictionary. o NZ SCoA-II.sav The following AMOS file is the confirmatory factor analysis input for the SCoA-II data set. o NZ SCoA-II.amw

SCoA-I Data Sets: The SPSS data file contains data information that was collected in 2003 from a national survey conducted to evaluate new mathematics test items as part of the Assessment Tools for Teaching and Learning Project. Items were split into 4 forms to

be completed by the students after they had finished the trial tests. There were no common items across the forms. This SPSS file contains all relevant variables and codes as outlined within this codebook/data dictionary. o NZ SCoA-I.sav The following AMOS files are the confirmatory factor analysis input for the SCoA-I data set. o o o o

NZ SCoA-IA.amw (Form A – negative/bad) NZ SCoA-IB.amw (Form B – useful) NZ SCoA-IC.amw (Form C – improvement) NZ SCoA-ID.amw (Form D – accountability)

4. Data Source Variables These variables provide a unique identification number for each participant and identify what form was filled out where applicable.

Summary Table Variable Code ID CoA_Form

Label Case identity number Form filled out

Variable Name: ID Description: A sequential identification number for each participant. Variable Name: CoA_Form Description: The form filled out by participant. In the SPSS data file for version 1: 1 = accountability items (Form D) 2 = improvement items (Form C) 3 = irrelevance items (Form A) 4 = validity items (Form B) Note. This variable is only in version 1.

5. School Measures These variables refer to school characteristics rather than individual participants. Not all versions contain this information, this is highlighted in the following section.

Summary Table Variable Code Schl_Type Schl_Gen_Type Schl_Region Main_Region Schl_Decile_Rate Schl_Decile_Grp Schl_Size Schl_Cluster Schl_Authority

Label School Type School Gender Type School Region School Main Region School Decile Rating School Decile Group School Size School Cluster School Authority

Variable Name: Schl_Type Description: Type of school where participant attended. Note. This variable is only in the SPSS data files for versions 5, 2 and 1. In the SPSS data file for version 5: 1 = Year 9 to 15 2 = Year 7 to 15 3 = Area In the SPSS data files for version 2 and 1: -1 = Missing data 1 = Primary/Intermediate 2 = Secondary 3 = Other Variable Name: Schl_Gen_Type Description: School type for secondary schools. In New Zealand secondary schooling there is a tradition of separate schools for boys and girls, often associated with higher academic status schools, such as grammar schools. Otherwise, most public schools are co-educational. Note. This variable is only used within the SPSS data files for versions 5, 2 and 1. In the SPSS data file for version 5: 0 = Co-educational 1 = Female 2 = Male

In the SPSS data file for version 2: -1 = Missing data 1 = Co-Ed 2 = Single Sex Boys 3 = Single Sex Girls In the SPSS data file for version 1: -1 = Missing data 1 = Co-Ed 2 = Single Sex Variable Name: Schl_Region Description: National region of participants’ school Note. This variable is only used in the version 5 SPSS data fie. In the SPSS data file for version 5: 1 = Northland 2 = Auckland 3 = Waikato 4 = Bay of Plenty 5 = Gisborne 6 = Hawkes Bay 7 = Taranaki 8 = Manawatu-Wanganui 9 = Wellington 10 = Nelson 11 = Marlborough 12 = Tasman 13 = West Coast 14 = Canterbury 15 = Otago 16 = Southland Variable Name: Main_Region Description: Main region that the participants’ school belongs to Note. This variable is only used in the version 5 SPSS data fie. In the SPSS data file for version 5: 1 = Upper North Island 2 = Lower North Island 3 = South Island From the above variable “Schl_Region” codes 10 to 16 belong in the South Island. Variable Name: Schl_Decile_Rate Description: School decile rating range from 1 to 10. Decile 1 is meant to indicate a school has children who come from the lowest 10% (decile) of the socio-economic stratum of NZ. The decile is assigned by the Ministry of Education in response to updated census information from Statistics New

Zealand. The index is determined by linking the addresses of children attending the school to the mesh block deprivation index for those addresses. Note. This variable is only used in the SPSS data file for version 5. In the SPSS data file for version 5: 1 = Decile 1 2 = Decile 2 3 = Decile 3 4 = Decile 4 5 = Decile 5 6 = Decile 6 7 = Decile 7 8 = Decile 8 9 = Decile 9 10 = Decile 10

Variable Name: Schl_Decile_Grp Description: School decile group, Low, Medium or High. The Low decile grouping included decile 1, 2, and 3. The Medium decile grouping included decile 4, 5, 6, and 7. The High decile grouping included decile 8, 9, and 10. Note. This variable is only used in the SPSS data files for versions 5, 2 and 1. In the SPSS data file for version 5: 1 = Low 2 = Medium 3 = High In the SPSS data files for versions 2 and 1: 1 = Low 2 = Low 3 = Low 4 = Medium 5 = Medium 6 = Medium 7 = Medium 8 = High 9 = High 10 = High Variable Name: Schl_Size Description: School size as categorized by small (