Professional Development in Assessment

“Professional Development in Assessment”

program

1

Conference program

Cover image: Selbymay; CC BY-SA 3.0

2

Welcome SIG-1 2014 It is our pleasure to welcome you to Madrid for the seventh biennial SIG-1 assessment conference entitled ‘Professional development in assessment: Challenges for teachers, schools and higher education programs’. Assessment has become one of the most important topics in educational research. EARLI SIG 1 invites researchers and practitioners to share their perspectives on how assessment affects learning, organizational change and assessment policy with colleagues at our biannual meeting. Main themes of the conference are formative and summative classroom assessment, large scale assessment and assessment policy. We are excited to have papers, symposia and round tables that present and discuss the latest topics in contemporary and prospective educational research on assessment from multiple perspectives. Program highlights are the keynotes of professor Cees van der Vleuten and professor Gavin Brown, and the newly created JURE award. We hope that this conference will be a great opportunity for you to start or continue developing a network within the assessment and evaluation field. We encourage you to seek opportunities to work collaboratively with other international scholars as we will be receiving presenters from all over the world. We encourage you to approach colleagues during the conference and establishing new links. Madrid is a beautiful city with many wonderful places to visit. Please explore http:// www.earli-sig1-conference.org/about-madrid.php for ideas. As organising committee, we wish you all a wonderful conference and a good time in Madrid! Katrien Struyven, Ernesto Panadero, Diana Baas and David Hidalgo Organizing committee

3

4

Organizing committee Katrien Struyven Ernesto Panadero Diana Baas David Hidalgo

List of reviewers Liesbeth Baartman Diana Baas Simon Beausaert Sue Bloxham David Boud Gavin Brown David Carless Jos Castelijns Brownen Cowie Theo Eggen Judith Gulikers Clair Hughes Anders Jonsson Renske de Kleijn

Eva Kyndt Sven de Maeyer Ernesto Panadero Ron Pat-El Frans Prins Marieke van der Schaaf Mien Segers Dominique Sluijsmans Kari Smith Jan-Willem Strijbos Katrien Struyven Gert Vanthournout Gerard van de Watering Bram de Wever

5

6

Program overview

7

8

Welcome

Table 1: Two-way Process in Science for All program- Ways to Improve the Quality of Teaching Process. Amira Rom & Michal Nachshon. Table 2: Developing a faculty specific framework for student evaluation. Inge Serdons & Iris Peeters. Table 3: Learning assessment in higher education. Orients to evaluate professional competences expressed in career’s graduate profile? Verónica Villarroel Henríquez & Daniela Bruna Jofré. Table 4: Implementing curriculum indepent testing in a post-initial bachelor Special Educational Needs and Remedial Teaching. Bram Pynoo, Hans Pluym & Inge Peeters. Table 5: Module assessment: content, standards alignment and grade integrity. Carmen Tomas , Graham Thomas & Jonathan Tepper.

16.00-17.30 Roundtables

15.30-16.00 Coffee

Scoping student teachers’ emergent perspectives of assessment literacy: Insights from three countries. Organiser: Frances Edwards; Chair/discussant: Susan Bloxham

Table 1: Application of Computer-Based Test to the Assessment of Reading Skills among Young Children. Andrea Magyar & Katalin Szili. Table 2: Using Design Based Research to Improve Assessment Literacy in Blended EFL Writing Instruction: Combining Learning Diaries and Feedback. Nuria de Salvador, María José Rochera & Ana Remesal. Table 3: Learning, assessment, and self-regulation in virtual environments: An exploratory study. David Hidalgo-Giménez & Jesús Alonso-Tapia.

Roundtables

Paper 1: Focusing Feedback: Enhancing the Effectiveness of Written Feedback. Carol Evans & Michael Waring. Paper 2: Feedback Comments in Higher Education: A Systematic Review of Evidence. Dai Hounsell. Paper 3: An interactional framework for feedback as a Paper 1: Chinese Students Teachers’ Beliefs about Assessdialogue: The role of (inter)personal factors. Jan-Willem ment. Junjun Chen. Paper 2: Developing assessment literacy in early career New Strijbos & Andreas Müller. Zealand secondary teachers: summative assessment. Frances Edwards. Paper 3: Use of feedback in Initial Teacher Education: Goal conflict and translation of feedback to feedforward. Mark Carver.

14.00-15.30 Symposium

12.30-14.00 Lunch Papersessions: Feedback

DOBLÓN

PATIO2

11.00-12.30 Keynote: Cees Van Der Vleuten

10.30

Breakout room 1

Main conference room

Table 1: Towards Assessment Reform: A Professional Development Initiative Connecting Research and Practice. Hilary Hollingsworth, Elizabeth Hartnell-Young & Marion Meiers. Table 2: Assessment of Teacher Self-Efficacy Beliefs and Concerns about Teaching of Graduating Preservice Teachers in New Zealand. David A.G. Berg & Lisa F. Smith. Table 3: Three contexts; three assessment dialects: What happens when assumed common notions of criteria, standards and formative assessment hit national contexts? Svere Tveit.

Roundtables

Paper 1: Assessing student learning outcomes at academic programme level: Creation of professional development activities for university teachers. Patrick Lai. Paper 2: In search for a stimulating and effective assessment training: first results from a case study. Anneleen Claassen & Steven Huyghe. Paper 3: Improving university exams through professional development of university teachers – Developing and evaluating a framework. Christoph Schindler, Manfred Prenzel & Tina Seidel.

teachers in assessment

assessment

ESCUDO

Breakout room 3

Paper 1: Students’ assessment conceptions in distance education. Michalis Michaelides & Pelagia Sidiropoulou. Paper 2: Assessment of Teaching Quality Based on Student Ratings – Chances and Limitations of Social Comparisons. Sylvia Rahn, Miriam Keune, Christoph Fuhrmann, Sabine Gruehn & Wolfgang Böttcher.

Papersessions: Professional development of Papersessions: Students’ conceptions of

COMENDADOR

Breakout room 2

Wednesday 27 august, 2014

12.30-14.00 Lunch

Paper 1: Skilful compliance or critical stance? Assessment literacy in academic communities. Birgit den Outer & Margaret Price. Paper 2: Assessing the assessment literacy: The effects of a training programme on lecturers perceptions. Victoria Quesada, Gregorio Rodríguez-Gómez & Marisol Ibarra-Sáiz. Paper 3: Examining the Assessment Literacy of External Examiners. Emma Medland. Paper 4: Assessment literacy for mixed-age teaching. Robbert Smit.

11.00-12.30 Papersessions: Assessment literacy

10.30-11.00 Coffee

Paper 1: The case for improving community processes for agreeing standards: investigating grading consistency amongst university assessors.Sue Bloxham & Birgit den Outer. Paper 2: Meaningful or Mock Transparency? Assessment standards in external examining. Margaret Price & Jane Hudson. Paper 3: ‘Double-blind peer review’: evaluating a community process for assuring academic standards in assessment. Mark Freeman.

Ensuring consistency in university assessment standards: the need for, and effectiveness of, academic community processes. Organiser/Chair: Sue Bloxham; Discussant: Clair Hughes.

Papersessions: Student characteristics, Paper 1: The Assessment of Underachievement in Secondary School Physics: Insights into Achievement Patterns and Gender Differences. Sarah Isabelle Hofer. Paper 2: How gender influences performance assessment: Teacher-student gender interaction in focus. Katarina Krkovic, Samuel Greiff, Sirkku Kupiainen, Mari-Pauliina Vainikainen, & Jarkko Hautamäki. Paper 3: Relationship between the English as a foreign language strategy use of elementary school students and their general learning characteristics. Anita Habók & Katalin Doró. Paper 4: Using large scale PIAAC assessment data to empower professional development: Problem-solving in the context of technology-rich environments. Raija Hamalainen, Bram De Wever, Antero Malin & Sebastiano Cincinnato.

Paper 1: Should teacher education be paying more attention to student teachers’ perceptions of formative assessment? Donna Hurford. Paper 2: Action Research for the Promotion of Assessment Literacy among Student Teachers. Maureen Rajuan. Paper 3: Undergraduate initial teacher education students’ peer assessment and feedback communities: a longitudinal study. Rita Headington. Paper 4: Assessing changes in teacher self-efficacy of preservice teachers over time. Franziska Pfitzner-Eden.

assessment and learning

Paper 1: The difficulty of item relevance in progress testing: the use of a rubric to appraise item relevanc. Xandra Janssen-Brandt, Arno Muijtjens, Dominique Sluijsmans & Erik Driessen. Paper 2: Learning progress assessment in reading: How can we construct and evaluate tests to monitor student reading progress? Natalie Förster & Elmar Souvignier. Paper 3: Context in mathematics examination questions. Jackie Greatorex.

development

Papersessions: Test and assessment

COMENDADOR

Breakout room 2

Papersessions: Assessment in teacher

education

Papersessions: Peer assessment

Paper 1: Structuring the Role of the Assessor and Assessee in the Peer Assessment Process: The Impact on Product Improvement and Peer Feedback Quality. Mario Gielen & Bram De Wever. Paper 2: Involving student assessors in direct-performance assessment: A qualitative content analysis of provided feedback. Ji-hye Kim, Jan-Willem Strijbos & Ingo Kollar. Paper 3: Effects of peer feedback content and sender’s competence on perceptions and mindful cognitive processing of written peer feedback. Markus Bolzer, Jan-Willem Strijbos & Frank Fischer.

DOBLÓN

PATIO2


Breakout room 1


Paper 1: Backwash Potentials: Reporting a Summative E-Assessment trial. Rebecca Hamer, Antony Furlong & Sarah Manlove. Paper 2: Technology-based assessment of problem-solving competence in VET – results from a pilot study. Rebecca Eigenmann, Christin Siegfried, Juergen Seifried, Eveline Wuttke, Kristina Koegler & Marc Egloffstein. Paper 3: Computer-based diagnostic assessment of thinking skills – the case of combinatorial reasoning. Attila Pásztor, Benő Csapó & Gyöngyvér Molnár.

Papersessions: E-assessment

Paper 1: Receptivity to Feedback: Research and Implications for Teacher/Pupil Interaction. Jeffrey Smith, Anastasiya Lipnevich, David Berg & Marg Kendall-Smith. Paper 2: Work Based Assessment: Effects of Feedback on Students’ Achievement in a series of Mini Clinical Evaluation Exercise (Mini-CEX). Mohd Nasri Awang Besar, Mohamad Nurman Yaman, Muhammad Arif Kamarudin, Tong Seng Fah, Saharuddin Ahmad & Maddalena Taras. Paper 3: Feedback giver: effects on feedback use and implementation. Anna Espasa, Teresa Guasch & Montserrat Martinez-Melo.

Papersessions: Feedback

ESCUDO

Breakout room 3

Thursday 28 august, 2014

9

10

19.00

Conference Gala Dinner

Paper 1: The Future of Self-assessment: A Core Skill Curriculum Instead of Another Assessment Activity. Gavin Brown & Lois Harris. Paper 2: Accuracy in Student Self-assessment: Directions and Cautions for Research. Heidi Andrade & Gavin Brown. Paper 3: The Future of Student Self-assessment: Known Unknowns and Probable Directions. Ernesto Panadero, Gavin Brown, & Jan-Willem Strijbos.

The future of self-assessment: Considered reflections on new directions. Organiser: Ernesto Panadero; Chair: Jan-Willem Strijbos; Discussant: David Boud


Paper 1: Student efficiency on online assessment tasks of foreign language vocabulary. Tibor Vidákovich, Tibor Vígh, Olga S. Hrebik & István Thékes. Paper 2: Comparative diagnostic assessment of German and English as a foreign language vocabulary of 11-12-year-olds. Tibor Vígh, Olga S. Hrebik, István Thékes & Tibor Vidákovich. Paper 3: Assessment of young learners’ English vocabulary with a diagnostic integrated test. Istvan Thekes & Anikó Németh.

development

Paper 1: Assessment of Learning and Teaching Through Quality of Classroom Assignments. Jeanette Joyce, Drew Gitomer & Charlie Iaconangelo . Paper 2: Benefits from Evaluation Data —Ways to Improve the Quality of Teaching Process. Amira Rom & Michal Nachshon. Paper 3: Around the world in A level mathematics. Jackie Greatorex, Jo Ireland, Stuart Shaw & Phineas Hodson.

Paper 1: Pupils’ conceptions of assessment in explicit and implicit classroom assessment practices. Ana Remesal. Paper 2: Different Perspectives on Teaching: How Do Teachers And Students Perceive Their Shared Lessons? Jessika Bertram, Sabine Gruehn, Sylvia Rahn & Wolfgang Böttcher. Paper 3: Student understandings of learning outcomes in teaching and assessment in higher education. Anton Havnes & Tine Prøitz.


assessment

Paper 1: Putting AfL to practice: the role of professional development and leadership practices. Diana Baas, Jos Castelijns, Mien Segers, Marjan Vermeulen & Rob Martens. Paper 2: The effects of formative assessment on self-regulated learning skills by sixth grade pupils. Kelly Meusen, Desirée Joosten- ten Brinke & Henny P. A. Boshuizen. Paper 3: Need for cognition and complex problem solving. Julia Rudolph, Katinka Hardt, Anja Strobel, Franzis Preckel, Mari-Pauliina Vainikanien & Samuel Greiff.

Papersessions: Assessment and instruction- Papersessions: Students’ conceptions of

al quality

assessment

Paper 1: A proposal for authentic assessment in the law curriculum. Rick Glofcheski. Paper 2: Does testing enhance memory in an elaborative educational context? Lydia Schaap, Peter Verkoeijen & Caroline Umuhoza. Paper 3: Using an assessment conceptual framework to facilitate institutional transformation of assessment. Carmen Tomas & Rachel Scudamore. Paper 4: Economic competence of teacher students - Results from a pilot study. Christin Siegfried & Eveline Wuttke.

assessment practices

Paper 1: The ‘assessment decisions’ framework: enhancing educators’ assessment practice. David Boud, Margaret Bearman, Philip Dawson, Sue Bennett, Matt Hall, Elizabeth Molloy & Gordon Joughin. Paper 2: Teachers’ use of differentiated assessment: the tiering model. Frances Wilson, Vikas Dhawan & Tim Gill. Paper 3: Effective Practice in Online Feedback - Communities of Practice. Ruth McAlister, Rachel Monaghan & Rosemary Moreland.

ESCUDO

Breakout room 3

Papersessions: Learning environments and Papersessions: Stimulating critical thinking

COMENDADOR

Breakout room 2

Papersessions: Supporting teachers’

DOBLÓN

PATIO2

15.30-16.00 Coffee

14.00-15.30

Breakout room 1



New approaches to the assessment of self-regulated learning. Organiser: Julia Klug; Chair: Ana Remesal; Discussant: Eduardo C. Cascallar.

Coffee

JURE awards and goodbye

14.00-16.00 Keynote: Gavin Brown

12.30-14.00 Lunch

16.00-End

Paper 1: Teachers’ evaluation in classroom practice. Anita Habók. Paper 2: Inquiring the origins of assessment conceptions to improve teachers’ assessment competence. Ana Remesal, Tomás Dominguez & Marc Lafuente. Paper 3: What does ‘good’ mean? – Exploring examiners’ interpretations of frequently used terms and qualifiers in assessment criteria. Sarah Manlove, Rebecca Hamer & Antony Furlong.

Paper 1: Using Adaptive Comparative Judgement to assess academic writing: reliability, psychological validity and misfitting judges. Tine van Daal, Vincent Donche, Sven De Maeyer, Liesje Coertjens, Marije Lesterhuis, Roos Van Gasse et al. Paper 2: Validation of a video-based online-tool to assess pre-service teachers’ professional vision: testing the generalizability across different teacher education tracks. Gloria Jahn, Kathleen Stürmer, Tina Seidel & Manfred Prenzel. Paper 3: Response Shift and the Validity of the Retrospective Pre-Test in Training Evaluation. Valentina Piwowar & Felicitas Thiel.

Papersessions: Peer assessment

Papersessions: Professional development of Paper 1: Context is all. Professional development in assessment does not work unless you take account of academics’ lived professional contexts. Lin Norton, Sarah Floyd & Bill Norton. Paper 2: Formative assessment and teaching practice: a call for educational research and teachers’ training. Serafina Pastore & Monica Pentassuglia. Paper 3: Continuing professional development in classroom assessment: contributions of a participative research based upon social moderation practices. Lucie Mottier Lopez, Walther Tessaro, Lionel Dechamboux, Fernando Morales & Sophie Serry.

Paper 1: Quality of assessment programmes: Assessors’ roles in quality assurance in practice. Frans Prins, Liesbeth Baartman & Raymond Kloppenburg. Paper 2: Why isn’t ‘knowing my students’ a good thing anymore? An exploration of benefits and dilemmas of anonymous marking. Rosemary Moreland & Isobel Hawthorne-Steele. Paper 3: What’s the point of moderation? Lenore Adie, Sue Bloxham & Clair Hughes.

teachers in assessment

ESCUDO

Breakout room 3

Papersessions: Quality of assessment in relation to assessor’s roles

assessment practice

Papersessions: Teachers’ perceptions of their

development

Paper 1: Examining undergraduate teacher assessors’ and assesses’ heuristics and their behavioural profiles in a webbased peer assessment context in science. Olia Tsivitanidou. Paper 2: Peer Assessment in a Large Introductory Class of Paper 1: Third wave on self-regulation measurement: when Biosciences - The use of Peer Assessment as an effective tool for learning. Viivi Virtanen, Henna Asikainen, Pekka Heino measuring is also an intervention. Ernesto Panadero & & Liisa Postareff. Sanna Järvelä. Paper 2: Self-Regulated Learning as a Process: Time-Series Paper 3: Anonymity within face-to-face peer assessment: Analyses of Diary Data from Low-Achieving Students. Sabine exploring the role of the teacher. Tijs Rotsaert & Tammy Schellens. Ogrin & Bernhard Schmitz. Paper 3: The PRO SRL Project - Assessing University Students’ SRL-Competencies in a multi-method-multi-informant approach.Julia Klug, Jöstl Gregor, Michaela Pichler, Barbara Schober, Christiane Spiel, Markus Dresel, Bernhard Schmitz & Albert Ziegler.


10.30-11.00 Coffee

Paper 1: Feedback in work-based learning: not working? Elizabeth Molloy & David Boud. Paper 2: Researching feedback dialogue using interactional analysis. Rola Ajjawi & David Boud. Paper 3: WatchMe: Workplace-based Assessment and Feedback by Means of E-portfolios. Marieke van der Schaaf, Jeroen Donkers, Bert Slof, Jan van Tartwijk, Erik Driessen, Atta Baddi & Olle ten Cate.

Researching new practices of feedback for learning in professional education. Organiser/Chair: David Boud; Discussant: Sue Bloxham.

COMENDADOR

Breakout room 2


DOBLÓN

PATIO2


Breakout room 1


Friday 29 august, 2014

11

12

Abstracts Keynotes

Symposiums

Papers

Roundtables

13


Photo: EARLI SIG1 Conference 2012

Keynote 11.00-12.30

Towards a future or programmatic assessment Cees van der Vleuten

Main conference room - PATIO 2

In the last 50 years the field of assessment of professional competence has seen remarkable progress. Developments in assessment technology have taken place across all areas of professional competence, ranging from cognitive to behavioural and emotional aspects of competency. This has been accompanied by extensive research. In order to make assessment more meaningful for learning, however, we need to change our thinking around assessment. We need to move from assessment of learning to assessment for learning, from individual assessment methods to a systems approach of assessment, from cross-sectional assessment to longitudinal approaches to assessment. This presentation will give an account of such a synthetic systems approach to assessment called programmatic assessment. This approach to assessment will be explained and illustrated with an existing assessment practice.

14

Wednesday 27 august, 2014 Symposium Scoping student teachers’ emergent perspectives of assessment literacy: Insights from three countries Organiser: Frances Edwards; Chair/discussant: Susan Bloxham

Initial teacher education is an important context for the development of the assessment literacy required by teachers if they are to respond to the diverse needs of today’s students. The focus on teacher assessment literacy can be seen as part of the current trend to focus on teacher quality. Teacher assessment literacy has implications for teachers’ ability to tailor their teaching in ways that optimise student learning. And, also the robustness and clarity with which they are able to sum up and report on student learning to parents and for accreditation/ exit qualification purposes. This symposium describes and analyses the experiences, understandings and development of student teacher assessment literacy in China, New Zealand and the England. The first paper scopes student teacher understanding of assessment purposes and practices in four universities in China. Findings

14.00-15.30 Main conference room - PATIO 2

highlight the role of examinations and signal the implications of the wider policy and societal context for assessment. The second paper traces student teacher development in New Zealand over 18 months, with a particular focus on summative assessment in secondary science. Participants attributed their growth in assessment capability to teacher education and school-based factors. The third paper elaborates on student teacher responses to feedback within their teacher education programme. It provides insights into the dynamic between the intended and unintended consequences of student teacher’s own experiences of assessment. Despite variations in the historical priority given to, for example examinations, the themes that emerged were similar. As a set, the papers highlight the complexity of assessment purposes, practices and consequences.

Paper 1 Chinese Students Teachers’ Beliefs about Assessment Junjun Chen

How student teachers conceive of assessment influences their classroom practice. This study investigated the responses of 531 first-year student teachers to a self-report “Beliefs About Assessment” questionnaire in six normal universities (teacher colleges) in the People’s Republic of China. The questionnaire, initially developed in the New Zealand context (Hill, Smith, Cowie, Gilmore & Gunn, 2013), framed around the principles of assessment for and of learning, beliefs about approaches and methods for assessment, and assessment rigour. It comprised 46 Likert-type items and five open-ended items. The statistical software package SPSS 20 was used to analyse the questionnaire using descriptive statistic and factor analysis. Factor analysis of the questionnaire yielded four factors. The first factor related to assessment is relevant and serves many purposes. The second factor focused on assessment having

negative effects (or being irrelevant). The third factor concerned the use of assessment to understand student learning. The fourth factor suggested that assessment is mainly for examinations and to prepare students for success. Thus, initial teacher education students on entry had a broad notion about the usefulness, relevance, and irrelevance of assessment for many purposes. They also demonstrated that the use of assessment is to understand students’ learning and prepare students for examination and ‘success’. However, what was missing was how teachers could use the results of assessment as a guide to appropriate teaching responses and what the role of students in classroom assessment might be. The results of this study provide a baseline for investigating whether the beliefs of Chinese student teachers change over period of their teacher education programme, and if so, how these changes have happened.

15

Wednesday 27 august, 2014 Paper 2 Developing assessment literacy in early career New Zealand secondary teachers: summative assessment Frances Edwards

This paper reports on an in-depth investigation into beginner secondary school teachers’ assessment literacy development during their participation in a one year graduate teacher education programme and on into their first six months in schools. Teachers working in New Zealand secondary schools fulfil a range of roles and responsibilities in common with teachers around the world. The qualification system used in New Zealand schools adds a layer of complexity: teachers not only have to prepare student for external examination but also have the responsibility for assessing students within a standards-based system. The results of both internal and external assessments contribute directly to students’ formal senior school and school-leaving qualifications. This presentation will review initial findings from an in-depth qualitative study of eight secondary teachers. Data comprised of interviews conducted at a number of points in an 18-month period, as

well as observations in university classes, a repeated survey, and the collection of a wide range of teacher-generated artefacts. The roles and influences of the teachers’ university teacher education lecturers and classes, their practicum experiences and mentor teachers, the school culture where they work, and their own personal views and experiences of assessment were all apparent, and contributed in a number of ways to their assessment literacy development. For each of the teachers, the formative use of all assessment types was a high priority for their practice. As the eight teachers told their stories, they attributed their changes and growth in this area to a number of factors. Their observations also exposed areas of personal challenge as they grappled with conflicting ideas. Specific examples will be provided which illustrate enablers and constraints in the development of their assessment literacy in the New Zealand secondary school context.

Paper 3 Use of feedback in Initial Teacher Education: Goal conflict and translation of feedback to feedforward Mark Carver

This paper will present initial findings from a mixed-methods study of over 300 undergraduate trainee teachers in a UK university. A problem was identified that trainees were unhappy with their feedback (including complaints that that it was late, insufficient, unhelpful, biased, or inconsistent). At the same time, staff did not believe that trainees made the best use of feedback, or even ignored it completely. The study started by looking at how trainees responded to their feedback, and then explored in more depth how this was related to different factors related to the student, the learning environment, and the feedback itself. This later stage involved interview and observation techniques based on the tacit nature of professional knowledge, such as using vignettes and repertory grids. A framework emerged in which feedback was better understood as serving multiple goals, so that a distinction can be made between feedback which is ‘for learning’ and

16

that designed for other outcomes, such as managerial or quality control. Analysis draws on the work of Becker et al. (1968), so that the behaviour of trainees (rather than self-report alone) is used as a basis for hypothesising about their beliefs and values. This is offered as an important shift in emphasis because it sees trainees not as using feedback incorrectly, but as behaving rationally based on the environment they perceive. It is therefore intended that this study will reveal fundamental changes which need to be made to the feedback environment experienced by trainee teachers in the UK, and how those changes need to recognise many of the different goals and unintended outcomes which relate to feedback. This project offers both a description of the feedback which can actually be evaluated as promoting learning, and an analysis of some of the specific barriers to that feedback being used effectively.

Wednesday 27 august, 2014 Feedback

Paper 1 Focusing Feedback: Enhancing the Effectiveness of Written Feedback Carol Evans & Michael Waring

This presentation focuses on maximising the impact of written feedback through the adoption of an holistic assessment design (Evans, 2013; Nicol, 2008). Much attention in recent years has been afforded to the development of sustainable assessment (Boud, 2000; Carless et al., 2011; Hounsell, 2007) highlighting the importance of the role and responsibility of students in the learning process if they are to be in a better position to self-assess their own work and therefore regulate their own learning. In assessing the efficacy of written feedback much attention has been focused on student engagement with feedback (Ball, 2010); issues of power in the relationship between student and lecturer (Croussouard & Pryor, 2009); tools to support understanding of the requirements of written assessment (Ramsay et al., 2007); access to feedback (Quinton & Smallbone, 2010). In this article we will argue the importance of holistic, integrated, and aligned designs in order to maximise the potential impact of written feedback (Fluckiger et al., 2009). This article describes three separate pedagogical in-

14.00-15.30 Breakout room 1 - DOBLÓN

terventions (cases) underpinned by the same theoretical framework to enhance students’ engagement with written feedback. The interventions have elements of an action research frame (Carr & Kemmis, 1986) involving the use of qualitative and quantitative methods. All the interventions took place in one UK university involving both newly qualified and experienced teachers undertaking postgraduate Masters study. In all three interventions the importance of attending to the following elements impacted on student engagement with written feedback: (i) student agency–central involvement of students in the assessment process; (ii) authenticity–relevance of assessment to the students’ working lives; (iii) explicit guidance–clarity about what constitutes good (Sadler, 2010); (iv) feedforward emphasis on what could achieved rather than what could not through the development of rubrics, integrated assessment opportunities, front-loading of formative assessment, and focused feedback techniques.

Feedback

Paper 2 Feedback Comments in Higher Education: A Systematic Review of Evidence

14.00-15.30 Dai Hounsell

Hitherto, empirical research into the comments made by university teachers on their students’ learning has not featured prominently in the very extensive efforts underway in many countries to investigate student discontent with feedback and to enhance its effectiveness. The study reported here seeks to remedy that history of relative neglect. Having traced over 100 studies of feedback comments in higher education at undergraduate, taught postgraduate and doctoral levels, a systematic review is now being undertaken. The chief goal is not to produce a conventional distillation of findings for which a high degree of universality could be claimed, but rather to be attentive to, and build the review around, the

Breakout room 1 - DOBLÓN

diverse and inescapable contingencies of feedback (i.e. non-trivial variations in terms of commenting purposes and modes, and in disciplinary and course contexts, in higher education settings). At the core of the analyses are four interwoven facets of feedback comments: functions or ostensible purposes, content (subject and task-specific), the language through which comments are communicated, and the overt or implied pedagogical goals. The present paper foregrounds the first of these, identifying functions along a spectrum ranging from validation to amelioration but also taking in interlocutory functions such as seeking further clarification, probing, alerting, proposing, question-raising, deba-

17

Wednesday 27 august, 2014 ting and conversing, and canvassing options. These should not be seen as providing a standard set of practical prescriptions but rather as an aid to reflec-

tion about what kinds of comments are likely to be most worthwhile in particular circumstances.

Feedback

Paper 3 An interactional framework for feedback as a dialogue: The role of (inter)personal factors

14.00-15.30

Jan-Willem Strijbos & Andreas Müller

In the past decade interest and research into the topic of feedback revived, marked by several reviews in close succession sharing the contemporary situated perspective on feedback, i.e. context and active processing by the recipient impact feedback efficiency. Despite their shared situated perspective, they predominantly conceive feedback as a unidirectional process from teacher to student. In contrast, recent conceptualisations of feedback practices reflect a transition from unidirectional to bidirectional, receptive to interactive, and monologue to dialogue. A broad review of feedback literature in education and workplace contexts was conducted to develop a conceptual framework for feedback as a dialogue. When linking the feedback process to dialogue, the four-sides communication model provides a very useful metaphor in that each (feedback)message contains four aspects: information, appeal, self-revelation, and relationship. Core feedback aspects, processes and mechanisms featuring strongly in (peer)feedback literature could be clustered along the four sides. Furthermore, conceptualising

Breakout room 1 - DOBLÓN

feedback as a dialogue implies that (inter)personal factors are integral to the conceptual framework. Indeed, attribution, self-efficacy and goal-orientation are well-known processes that affect feedback reception; but there is hardly evidence as to whether, how and to what degree these processes influence feedback composition and provision. Moreover, when the sender composes and provides a feedback message (or a recipient receives feedback) his/ her representation of the recipient (or sender) is simultaneously activated and influences the composition and provision (or reception) process. The representations are in turn shaped by (a) formal status in relationships, (b) informal status qualified by competence, and (c) friendship and/or anonymity. In sum, a conceptual framework could be developed for feedback as a dialogue on the basis of a broad literature review. Substantial evidence illustrates the feasibility of the conceptual framework and its central premise that (inter)personal factors play a major role in feedback as a dialogue.

Professional development of teachers in assessment Paper 1 Assessing student learning outcomes at academic programme level: 14.00-15.30 Creation of professional development activities for university teachers Breakout room 2 - COMENDADOR Patrick Lai

Assessing the academic and generic competence of students in academic programmes is crucial in determining the quality of programmes in universities. A review of the literature indicates that there are resistances among academic staff in developing rubrics for programme assessment. This phenomenon might be partially attributed to the fact that there are qualitatively different ways by which academic staff understood the concept of grade descriptors (Tan and Prosser 2004).

18

The significance and originality of this paper is that it has extended the literature of use of rubrics in higher education by exploring into the area of professional development strategies to support teachers in university to make use of valid and reliable rubrics for assessing student learning outcomes at the programme level. Seven academic programmes, which ranges from hard sciences to humanities, accepted the invitation to participate in this study. The project team con-

Wednesday 27 august, 2014 ducted a systematic process of collecting and collating feedback from participating programme teams throughout the project. This involved keeping notes from ongoing observations, interviews and working group meetings with participants at strategic times during the project development. A final round of individual post-project interviews was held with key participants representing the respective programme team. It is clear from the experiences of this project that the successful assessment of institutional outcomes at the programme level begins with and is

dependent upon careful planning at programme and particularly subject levels, which include several important steps. In determining and refining the professional development strategies, several steps were considered. These would include (1) defining learning outcomes at the institutional level; (2) identification of possible assessment tasks within academic programmes; (3) generation of rubrics of learning outcomes; and (4) training of assessors to use the rubrics for assessment. Details of these will be addressed in the paper.”

Professional development of teachers in assessment Paper 2 In search for a stimulating and effective assessment training: 14.00-15.30 first results from a case study Breakout room 2 - COMENDADOR Anneleen Claassen & Steven Huyghe

Assessment continues to be high on the agenda in Higher Education as many research findings confirm the impact on students’ learning. Despite this evidence, the assessment practices of teachers seem difficult to change or improve. In this paper we present a professionalization activity of the Educational Development Unit (EDU) at KU Leuven in which new faculty staff are guided in translating the research findings on assessment into their concrete teaching practices. Throughout the years this training has had several designs to better fit the assessment needs of the new faculty, and to be more effective in stimulating good assessment practices.

The most recent iteration was to implement a modular format. Key elements of this professionalization format are an interactive session at a 2-day kick-off workshop and an accompanying module. This module starts with a task to prepare for a workshop of half a day, followed by an assignment at the own workplace, and a closing reflection session. The format of the training and the first reflections on this approach will be explained in detail. By outlining this example we want to argue that the presented approach is effective to help new faculty members reflect upon and improve their assessment practice(s).

Professional development of teachers in assessment Paper 3 Improving university exams through professional development of university 14.00-15.30 teachers – Development and Evaluation of a framework Breakout room 2 - COMENDADOR Christoph Schindler, Manfred Prenzel & Tina Seidel

Exams strongly influence student learning and the quality of acquired qualifications. University teachers are the main agents in developing exams. Therefore, professional development of university teachers is an effective means to improve the quality of exams. Effective professional development programs focus on real and relevant problems in everyday practice. Only few data is available on the assessment practice of university teachers, the problems and challenges they encounter and the pro-

blem areas and weaknesses in university exams. In this study six key building blocks for test development are derived from educational measurement literature that serve as the cornerstones of a professional development program. The goal of the study is to evaluate whether these building blocks capture the problem areas in existing exams and address the challenges university teacher encounter in developing and administering exams. Empirical investigations are conducted in the context of the profes-

19

Wednesday 27 august, 2014 sional development program. The sample consists of 14 university teachers from the Department of Mechanical Engineering (n=11) and Department of Math (n=3) at the Technical University Munich. Data sources are the existing and newly developed learning goals (N=57) and exams (N=16) with corresponding Items (N=598) developed by the participants. In addition, verbal data is being recorded during work meetings (N=24). The results provide data on the characteristics and problems of existing exams, on changes made during the professional de-

Paper 1

velopment program and problems and challenges university teachers encounter in developing exams. Furthermore, inferences are drawn on the suitability of the proposed building blocks to serve as framework for professional development programs focusing on the improvement of university exams. In conclusion, this study contributes knowledge that can inform the design and purposive selection of contents to be included into professional development programs aimed at improving university teachers´ capacity to develop high-quality exams.

Students’ conceptions of assessment Students’ conceptions about assessment in open 14.00-15.30 distance education programs Breakout room 3 - ESCUDO Michalis Michaelides & Pelagia Sidiropoulou.

Student conceptions toward assessment relate to how students approach learning and studying. Previous research has identified four conceptions: assessment as improving learning and achievement, as a means for making students accountable, as being enjoyable and as irrelevant to learning (Brown & Hirschfeld, 2008). Assessment and evaluation, as well as the overall educational experience, in open, distance higher education institutions are different from conventional tertiary institutions. The purpose of the present study was to examine students’ conceptions about assessment in open, distance education programs. An online survey was administered to students in an open university in Cyprus. Two hundred and

sixty five students responded to a Greek adaptation of Brown’s (2011) Students’ Conceptions of Assessment VI inventory. The notion that assessment is a process for improving teaching and learning was more strongly endorsed. Support was given for the conception of assessment as an enjoyable activity and as a means for holding students accountable for their learning. Participants expressed disagreement on items stating that assessment is irrelevant to their education. Students’ conceptions about their assessment in open, distance, higher education have not been extensively studied. Findings inform existing assessment practices and could guide the development of new assessment procedures tailored to open, distance education.

Students’ conceptions of assessment Paper 2 Assessment of Teaching Quality Based on Student Ratings – 14.00-15.30 Chances and Limitations of Social Comparisons Breakout room 3 - ESCUDO Sylvia Rahn, Miriam Keune, Christoph Fuhrmann, Sabine Gruehn & Wolfgang Böttcher

It is highly recommended to teachers and schools to use student ratings to measure and develop teaching and education quality. In this context it is common to assess teaching quality by comparing the feedback a teacher gets from his students with the average of a sample of student ratings from different schools and subjects. Although this procedure is in general use its fairness can be doubted especially for

20

two reasons. Firstly Students are not interested in the topics of different subjects to the same extent, and secondly student ratings of teaching quality may be influenced by their interest in the topic of the lesson. Therefore the presentation is going to discuss to which extent student ratings of teaching quality are biased by the interest profile and the learning motivation of their students. Furthermore it is going


Photo: Cibeles Fountain, Madrid. Turismo Madrid. CC-by-nc-sa 2.0

to be discussed in how far the student ratings in different subjects are affected by these factors. The argumentation is based on a sample of 2212 students from 49 academic high schools (including 1328 students from 37 professional secondary schools) which evaluate the quality of teaching in Mathematics, German, Economy, Pedagogy and Electrical Engineering. Referring to the results of multivariate statistical analysis and observing the multi-level structure of data, it is going to be shown in two steps that, firstly

the interest and motivation to learn differs significantly between the subjects and in interaction with the type of academic high school. Secondly student ratings of teaching quality are affected by the interest and motivation to learn in all subjects but are not biased in every subject in the same way. The lecture is going to lead to a discussion of practical implications of the results and emphasis possible strategies which take the chance of social but avoid the danger of unfair comparisons.

21

Wednesday 27 august, 2014 Roundtables

Table 1 Two-way Process in Science for All programWays to Improve the Quality of Teaching Process

16.00-17.30

Amira Rom & Michal Nachshon

In this study we present the impact of the evaluation process on the development and decision-taking processes involved in designing the Science for all curriculum. Science for all students are a heterogeneous population including students of high schools and in technological trends, who do not specialize in the sciences. Science for all teachers are university educated with degrees in exact sciences, who have participated in continuing in service training programs for teaching in Science for All. The Science for All evaluation method is multi-dimensional, comprising a range of evaluation instruments that reflect students’ diverse abilities. The evaluation checks both the processes taking placing throughout the learning period and its outcomes. Assessing the outcomes enables the learning to be evaluate and also enhances the teaching–learning process. The assessment process reflects evaluation for

the benefit of learning. The portfolio, which was chosen as a central tool for assessment, is a process which demonstrating the student’s learning process, achievements, progress, knowledge and ability. The Science for All portfolio is a central tool that provides a realistic picture of the student’s performance and reflects the quality of the teacher’s work. It enhances the learning and enriches the Science for All teaching processes. Over the past seven years, at the end of each school year, a sample evaluation process was conducted in Science for All subject. The portfolios were evaluated by a group of trained teacher-evaluators. The aim was to analyze the quality of teachers’ work through their students’ outcomes. The sample evaluation was a significant factor in enabling following up of the teacher’s work-style and in enhancing the teaching– learning–assessment process.

Roundtables

Table 2 Developing a faculty specific framework for student evaluation Inge Serdons & Iris Peeters

The Faculty of Science, KU Leuven has developed a hands-on tool that enables program committees to develop and implement their own context-specific vision on student evaluation. The main objective of this vision is to use evaluation as a tool with multiple benefits, namely activating students at an early stage during their learning process, giving feedback to students in order to optimize their learning paths and of course evaluating the extent in which the students have reached all the objectives of the program. In order to reach these goals, the Faculty of Science encourages continuous assessment where, depending on the specific context, the evaluation during the semester can be either formative or summative. Furthermore, the program committees are asked to introduce a quality assurance on the eval-

22



uations. This demands a change in mentality of the professors who are traditionally used to being the sole person responsible for evaluating their students. The vision of the Faculty of Science has been translated into a hands-on tool that is currently being used by individual programs to create their own specific plans for student evaluation within the above mentioned framework. In the next phase, a SWOT-analysis of the current situation will allow to identify action items to optimize the evaluation process in every program. The final goal is to implement a specific view on student evaluation in all educational programs of the Faculty of Science, hereby guaranteeing that every student will be evaluated in a qualitative and structured manner throughout his or her curriculum.

Wednesday 27 august, 2014 Table 3 Learning assessment in higher education. Orients to evaluate professional competences expressed in career’s graduate profile? Verónica Villarroel Henríquez & Daniela Bruna Jofré

A topic of discussion in undergraduate training has been the kind of learnings students achieve once they graduate. For many years higher education was centered on information delivery, without concerning about how each student integrated, synthesized and applied knowledge to personal, social and work life. The attention about the quality of teaching and pedagogical processes, derive in efforts to renew educational methodologies. In different parts of the world, this was traduced in the adoption of the competence model for students training in higher education. One weakness in the application of this approach it´s related to the learning assessment, tough the formats of the tests are relevant for the teacher, but not outside of the classrooms. The evaluations that are traditionally used at college, are oriented to measure data


memorizing, instead of construction and integration of knowledge. It has been privileged the measurement of memoristic abilities, over problem based learning, project methodology, and the creation of group products that have value in the working world. In Chile, classroom learning assessment is the most deficient area in the pedagogical practice, according to the Chilean teacher performance evaluation in the last years. The worst evaluated dimensions have been the quality of the evaluation instruments used by teachers, and the use of the information provided by these. This study aims to know and evaluate the congruence between the evaluation methodologies and instruments, and the graduate profile declare on the curricula of 6 higher education careers.

Table 4 Implementing curriculum indepent testing in a post-initial bachelor Special Educational Needs and Remedial Teaching Bram Pynoo, Hans Pluym & Inge Peeters

In this contribution the rationale for redesigning the evaluation method of the post-initial bachelor Special Educational Needs and Remedial Teaching is discussed. Currently, students are confronted with often overlapping evaluations per course, which is a burden for both students and teacher educators. In the context of the course “Integrated assignment” (GIO), we developed a curriculum independent test (CIT) to assess students knowledge and competences in an integrated manner. The GIO is a case study of a toddler, elementary or secondary school child

Roundtables

Roundtables 16.00-17.30 Main conference room - PATIO 2

with special educational needs for which students have to propose - in groups of 5 to 7 students - a program tailored to the needs of this child. The CIT is devised based upon the competences students need to attain at the end of their study. This CIT will be used by the jury to evaluate the students. The CIT is complemented with peer evaluations of the group work. At the SIG EARLI meeting, we will propose the CIT that is currently being developed and relate the scores of the students on the CIT to the grades students obtained for the regular courses.

23

Wednesday 27 august, 2014 Roundtables

Table 5 Module assessment: content, standards alignment and grade integrity Carmen Tomas , Graham Thomas & Jonathan Tepper

An exploratory exercise considered the design of exams in modules and alignment. Interviews with six module leaders in different disciplines revealed good practices around exam design. However, the discussions also elicited potential threats to grade integrity. These are categorised as: • lack of transparency in the alignment of learning outcomes and standards; • unassessed or sparse assessment of some learning outcomes in comparison to others; • exam design, specially where optional questions exist, was found to pose some challenges to grade integrity as questions some times were not found to be of same level of difficulty; • extraneous factors to the exam design, such as organisation of the delivery in tutori-

Andrea Magyar & Katalin Szili

24


als and previous coverage of content, were equally seen to challenge grade integrity. Overall, the small scale study has revealed that professional development around assessment design and alignment of standards needs to play much closer attention to: • the relation between learning outcomes and definition of criteria and level descriptors • the relation between assessment design and delivery factors (e.g. previously seen or rehearsed questions as opposed to unseen questions) • procedures and guidance to support checks, perhaps at design stages, of alignment and due coverage of learning outcomes in the assessment.

Table 1 Application of Computer-Based Test to the Assessment of Reading Skills among Young Children Evaluation of the measurement of student performance is one of the most powerfully the developing area of pedagogical research. The computer-and web-based education systems with integrated test systems are widely spread. The most common form of the technical based assessment is the Computer Based Testing. In its simplest form the traditional paper-based tests are digitalised without changes. In this case the linear order of the exercises remains the same as in the original form, only the test medium changes. We used this form in our research to illustrate the benefits of computer-based environment. During the research, the answer to the following questions were asked: (1) Can computer-based testing be applied to the assessment of young children; (2) is it an effective tool among 2-6-grade children; (3) is there any difference in the operation of the test

16.00-17.30

Roundtables 16.00-17.30 Breakout room 1 - DOBLÓN

in different grades. In the pilot sample 112 primary school pupils were involved. The measurement tool was the digitalised version of a diagnostic vocabulary-measurement test. The test was constructed and delivered by the eDia platform and the online data collection was carried out via internet by using computers facilities available at schools. The results showed that the online test was suitable for the measurement of vocabulary of young children. The reliability of the test was eligible (Cronbach alpha=0.963). The test could well differentiate among grades, the children’s performance were between 46% and 93%. The research was an example of how traditional paper and pencil based tests can be transformed into effective computer-based measurement tool that could be easily applied for the assessment of young children.

Wednesday 27 august, 2014 Table 2 Using Design Based Research to Improve Assessment Literacy in Blended EFL Writing Instruction: Combining Learning Diaries and Feedback Nuria de Salvador, María José Rochera & Ana Remesal

We present a Design Based Research (DBR) experience to improve teacher assessment literacy in the context of teaching writing in an EFL, blended environment, in secondary education. This experience grounded on an assessment design based on a wiki tool, where writing tasks, learning diaries and public e-feedback were combined to monitor and improve students’ learning. It followed a two-cycles model, with four phases of development: (i) defining the problem (improving students’ English writing skills); (ii) developing solutions theoretically grounded (designing an assessment program based on the blended use of ICT for providing public feedback and promoting individual reflection); (iii) implementing the instructional design in the real classroom, and collecting empirical evidences (students evaluation questionnaires); (iv)


analysing data (descriptive statistics), and formulating design principles to prepare the second cycle of implementation. In this paper, we present the first cycle. Data consisted in online students’ writing productions, teacher’s feedback and students’ reaction to it in learning diaries. Students filled in satisfaction and evaluation questionnaires, and the teacher kept a log. All these instruments provided empirical evidences of the strengths and weaknesses of the instructional design in general, and its assessment program in particular. The reflective nature of DBR and the characteristics of this research supported awareness on practical and theoretical issues related to the quality criteria of formative assessment and professional development.

Table 3 Learning, assessment, and self-regulation in virtual environments: An exploratory study David Hidalgo-Giménez & Jesús Alonso-Tapia

The purpose of this study was to explore the impact that on-line resources have on motivation, self-regulation, and learning of Secondary Education students. Resource design and development influence learning; hence, educators could enhance their teaching and assessment practices by offering students help through technology. Therefore, it seems necessary to explore new ways to support teachers with regard to the organization of instructional sequences and effective assessment procedures. With this purpose, before the implementation of a virtual environment developed for this study, student motivation, text comprehension, was evaluated using standardized tests between a sample of 104 Spanish Secondary School students. The participants were assigned randomly to one of the eight treatments namely, Text (T), Text, multimedia (TM), Hypertext (H), Hypertext, multimedia (HM), Text, self-assessment (TS), Multimedia,

Roundtables

Roundtables 16.00-17.30 Breakout room 1 - DOBLÓN

self-assessment (MS), Hypertext, self-assessment (HS) and Hypertext, multimedia, and self-assessment (HMS). Subjects read the same on-line information about several topics in different formats, depending on each treatment, and answered some comprehension and application questions. After implementation of the program self-efficacy, reading comprehension, immediate response, later knowledge, generated interest and perceived usefulness was evaluated. From this integrative approach, the effectiveness of multimedia, understood as the student achievement in each condition, was explored. Preliminary results indicate that virtual environments offer a variety of sources to develop instructional, assessment tools and tasks that promote self-regulatory learning, especially in contexts when feedback is provided and application of knowledge is requested. Also, assessing effectively through virtual environments will be discussed.

25

Wednesday 27 august, 2014 Table 1 Towards Assessment Reform: A Professional Development Initiative Connecting Research and Practice Hilary Hollingsworth, Elizabeth Hartnell-Young & Marion Meiers

In recent years research has determined the undeniable impact of assessment on student learning and presented implications for assessment practice. Attempts to build assessment knowledge and capacity across education communities have taken different forms, including assessment-focused workshops, courses, conferences and publications. However, the translation of research ideas into practice for many teachers, schools and systems remains a challenge. This paper reports a strategic initiative currently being implemented in Australia that focuses on strengthening assessment literacy in teachers, schools and the wider education community. The initiative has several design features that make it unique and anticipate its success.

David A.G. Berg & Lisa F. Smith

26

16.00-17.30 Breakout room 2 - COMENDADOR

Propelled by a major review of the assessment research, the initiative has at its core the articulation of a clear, fundamental purpose for assessment and the goal of realising the enactment of this purpose by teachers, schools and policymakers. A key component of the initiative design is a coherent, ongoing program of research-based opportunities for professional development that are intricately linked to classroom practice. The program is based on a set of sound principles of professional development and targets participants at all levels of education. This paper will present details related to the initiative and report initial insights and findings related to the professional development program that is its focus.

Table 2 Assessment of Teacher Self-Efficacy Beliefs and Concerns about Teaching of Graduating Preservice Teachers in New Zealand This study compared the assessment of graduating preservice teachers’ teacher self-efficacy beliefs with their concerns about teaching. The purpose of the study was to establish the degree that these two constructs might be correlated, and if one might act as a proxy for the other. A second goal was to explore how teacher educators might assess difficulties perceived by preservice teachers, thus improving their preparation for classroom teaching. Teacher self-efficacy beliefs are teachers’ beliefs about their capacity to carry out their professional roles successfully. These have been shown to relate to significant positive outcomes for students, schools, and teachers (Tschannen-Moran, Hoy, & Hoy, 1998). Teacher self-efficacy beliefs have been linked with a related construct: teachers’ concerns about teaching (Boz & Boz, 2010; Dunn, Airola, Lo, & Garrison, 2013). Ameliorating concerns about teaching have been recognized as central to preservice teacher education (Dunn & Rakes, 2010), and have been shown to be relatively stable over time (Cherubini, 2009).

Roundtables

Roundtables 16.00-17.30 Breakout room 2 - COMENDADOR

In this study, graduating primary preservice teachers from New Zealand (n = 47) completed the Teachers’ Sense of Efficacy (long form) (Tschannen-Moran & Woolfolk Hoy, 2001) and the Concerns About Teaching Scale (Smith, Klein, & Mobley, 2007; modified with permission), during the final week of their programmes. Exploratory factor analyses were used to group items for each measure and compare the resulting subscales to those established for the measures. Some correlations resulted, most notably between a TSES subscale comprised of items relating to efficacy for the use of classroom “survival” strategies and a CAT subscale that comprised items relating to teacher stress. However, the main finding was that that the constructs measured by these instruments are unique. The results are discussed in terms of the need for teacher educators to assess both constructs to increase the likelihood preservice teachers’ success in the classroom.



Table 3 Threecontexts;threeassessmentdialects:Whathappenswhenassumedcommon notionsofcriteria,standardsandformativeassessmenthitnationalcontexts? Svere Tveit

When educators (in primary, secondary and tertiary education) researchers and policy makers talk about assessment we employ language that usually stem primarily from the national context each and one of us come from and/or work in. A second source of language that feed into our national languages with increasing strength is international scene, whether from books, journals, conferences or other modes of sharing experiences from research and practice. This paper explores three education contexts – Norway, Sweden and Queensland (Australia) – that share a strong emphasis on teacher judgments for high-stake assessments, albeit in distinct different ways, and discusses how terminology is used differently across contexts. The paper first provides a literature review identifying how the three national/state contexts of educational assessment in secondary education developed post World War II and how the teaching profession

Roundtables 16.00-17.30 Breakout room 2 - COMENDADOR

and associated disciplines have shaped the nations’ assessment language. Second, by analyzing contemporary policy documents it explores how key concepts from the growing international research literature on educational assessment are being translated into the local contexts. Assessment criteria, assessment standards and formative assessment are among key terms and concepts investigated. The paper identifies indistinctive application of these terms within national contexts and discusses the complexity of understanding the concepts across national or state contexts. The paper concludes by identifying typical pitfalls in nation’s attempts to borrow policy from each other and raises questions as to the practical implications for teachers and students when educators, researchers and policy makers’ fail to understand each other’s assessment dialect.

27

Thursday 28 august, 2014 Symposium Ensuring consistency in university assessment standards: the need for, and effectiveness of, academic community processes Organiser/Chair: Sue Bloxham; Discussant: Clair Hughes

Internationally assessment standards for higher education are increasingly being defined (eg, Tuning in Europe, QAA in UK and OLT in Australia). However, we do not know whether lecturers can consistently apply such standards in student assessment. Research suggests that individuals’ standards differ considerably. Explanations of this variation are located in socio-cultural and social constructivist perspectives on academic judgement suggesting a vital need to develop greater consensus. This symposium explores this issue of shared standards and the potential of community processes for developing consensus. The first reports assessors’ inconsistency as indicated through a study of the constructs examiners use in judging assignments. Few clear patterns emerged within or between the examiners in the disciplines involved. Where similar constructs were shared they were interpreted differently, appearing to result in manifestly different standards. The second explores potential reasons for this inconsistency in standards which exists despite the systems and processes in place that aim to achieve

Paper 1


transparency and consensus. Using data about influences on examiners’ conceptualisations of assessment standards two key positions are identified. These are critiqued in relation to consensus and an alternative approach founded in academic community processes is advocated. The third reports an attempt to operationalise a national model for developing, sharing and assessing standards in one discipline by drawing on accepted processes for publishing research. Academics (and some practitioners) engaged in collaborative processes to develop threshold learning standards for accounting programmes. Subsequently external assessors double-blind peer reviewed random samples of student work and the suitability of assessment tasks. These activities reduce assessor variation and improve assessment literacy and validity. This symposium will provide an opportunity for informed debate around how shared understanding of assessment standards can be progressed in order to improve consistency, confidence and transparency in those standards.

The case for improving community processes for agreeing standards: investigating grading consistency amongst university assessors Sue Bloxham & Birgit den Outer

Higher education systems use processes such as moderation, criteria and scaling aimed at securing academic standards and equalising differences between assessors. They rest on an assumption that individual assessors’ judgements are derived from a shared view of quality in the relevant field although assessors’ inconsistency and unreliability are well documented in research. The project reported here sought to investigate the similarities and differences in the standards used by experienced assessors by exploring the constructs (criteria) they use in making judgements. It employed Kelly’s (1991) Repertory Grid (KRG) method to collect data about the cognitive activity of expe-

28

rienced assessors, facilitating assessors to articulate the nuanced constructs they use in distinguishing between pieces of student work. Twenty-four experienced assessors in four disciplines from twenty UK universities were provided with five borderline examples of student work. The KRG exercise elicited the constructs used to judge these assignments based on an in-the-moment evaluation of actual student work, rather than espoused practice. Data analysis attended to the range of constructs, ranking of constructs by importance, shared constructs across a discipline, consistency of scoring within each construct, and consistency of overall judgement for each piece. Few clear patter-

Thursday 28 august, 2014 ns emerged within or between the four disciplines. Where apparently similar constructs were shared, they were interpreted differently, appearing to result in manifestly different standards. Assessors’ overall judgement of the quality of student work revealed considerable divergence in judgement. Even where the overall judgements about an assignment were similar, assessors frequently

made different judgements about the strengths and weaknesses of particular aspects of the work. These findings indicate that shared language does not ensure shared interpretation of common criteria in judging complex academic work and, thus, there is a firm case for community processes which encourage sharing of both holistic assessment and the meanings of individual criteria.

Paper 2 Meaningful or Mock Transparency? Assessment standards in external examining Margaret Price & Jane Hudson

Introduction of new subjects and types of programmes, increasing collaborations, decreasing resources per capita and internationalisation have dramatically altered the global higher education landscape. These changes have led to calls for greater comparability and public accountability with regard to academic standards, both at international and national level. An example of where these calls have been addressed through various inquiries is in the UK external examining system, the system relied upon to safeguard and maintain consistency of standards across institutions. Improvements to the external examiner system have concentrated on examiner recruitment, procedures and stakeholder understanding of examining, i.e. changes to making processes more transparent, with the aim of creating greater institutional accountability. However, there is little understanding of how standards in academia are conceived, constructed, and applied across institutions; the complex nature of sharing standards has been elided in much of the literature.

Funded by the Quality Assurance Agency and the Higher Education Academy the research reported on is a mixed-method qualitative study in which 24 external examiners in four disciplines, from 20 diverse Higher Education institutions took part. It explores how examiners negotiate standards at a time when HE institutions are increasingly compelled to adhere to principles of transparency and accountability through systems aimed at making standards explicit. It concludes that taking away risks in the name of a, predominantly, tick-box transparency and accountability poses a new risk, which is rendering the system meaningless with regard to maintaining a sense of national standards. By unpicking what external examiners do, and how they talk about standards, the paper aims to locate the place the external examining system could occupy as custodian of academic standards. A vision for a new, meaningful transparency in academic standards firmly located within community processes is proposed.

Paper 3 ‘Double-blind peer review’: evaluating a community process for assuring academic standards in assessment Mark Freeman

This paper reports an attempt to operationalise a national model for developing, sharing and assessing academic standards across Australian higher education providers, contributing to calls for reliable and valid approaches to assuring graduate outcomes. Policy developments in this field have focused on creating explicit descriptions of standards through,

for example, qualifications frameworks and benchmarking discipline standards. Yet, it is clear that explicit standards alone are insufficient as a basis for assuring comparability of standards (Moss & Schultz 2001). Various studies emphasise the interpretive, personalised and subjective nature of standards, learnt through active participation in relevant

29



communities and practices and lacking the characteristics that would enable a broad and enduring consensus (Bloxham & Price 2013). This paper reports an approach to assessing standards involving consensus moderation workshops. Many accepted practices employed to maintain the standing of research journals are mirrored in assessing standards using a consensus moderation approach (Sadler, 2013), raising credibility and facilitating acceptance. This process involves pre-work where academics independently review a sample assessment task and samples of student work. Judgements and comments are entered online. Anonymous results are published for reviewer interrogation. At workshops reviewers first reach consensus in small groups. Groups then simultaneously reveal

30

judgements, followed by argument and defence as key differences are discussed. Evaluation includes experimental and survey research. Workshop participation appears to assist academics to tune judgement-making ability since the standard deviation of marks halved for the treatment group compared to a control group showing no reduction. Surveys reveal that significant professional development at workshops occurs around assessment literacy and design as well as conceptions of standards (Watty et al. 2013). Sharing lessons learned has improved the standards-setting process, saving money and mistakes, assisted engagement and dissemination, and consistent with Price et al. (2008), triggered dialogue about assessment. References available with the extended summary.

Thursday 28 august, 2014 Paper 1 Structuring the Role of Assessor and Assessee in Peer Assessment Process: Impact on Product Improvement and Peer Feedback Quality Mario Gielen & Bram De Wever

This research focuses on the added value of structuring the role of both the assessee and the assessor in peer assessment (PA) processes in a wiki-based learning environment. First year university students, enrolled in an educational sciences program were asked to collaborate in small groups and give feedback on a peer’s draft version of an abstract of research article in a wiki environment. Two subsequent studies were set up. The main aim was to study the product improvement (students were asked to sequentially write and revise three draft versions) and the quality of students’ feedback (students received feedback from a peer for each of the draft versions). In the first study, the focus was on the degree of structure ifeedback forms and three conditions were implemented: a non-structured peer feedback condition, a basic structured peer feedback condition and an elaborate structured peer feedback condition. For the second study, the same form was


used for all groups, but the focus was on how the role of the assessee and the assessor was additionally structured through respectively a feedback request list and/or a content check list, which resulted in four conditions: a no request - no content condition, a request – no content condition, a no request – content condition and finally, a request – content condition. Results of the first study showed that the wiki product improved significantly in the initial phase, but this effect decreases over time. Especially, product improvement was significantly higher for the elaborate condition. From abstract 1 to abstract 3, peer feedback quality augmented over time for all conditions, but the effect of the elaborated structure condition surpassed the two other conditions. For the second study, it is hypothesized that the request – content condition will outperform the other conditions regarding product improvement and peer feedback quality.

Paper 2 Involving student assessors in direct-performance assessment: A qualitative content analysis of provided feedback Ji-hye Kim, Jan-Willem Strijbos & Ingo Kollar

The viability of involving student assessors in direct-performance assessments is highly dependent on their capacity to identify and interpret specific aspect of the performance and their competence in providing quality feedback. This implies that student assessors should have a certain degree of (a) prior-knowledge in the domain, and (b) feedback provision skills. However, it is unclear to what extent student assessors’ prior-knowledge affects the quality of feedback they provide to fellow student during direct-performance assessment. This case study explores the viability of involving student assessors in such assessment and the impact of prior-knowledge on the feedback quality. Participants were four senior students in the third semester of a Learning Sciences master’s program, who assessed the performance of 25 junior first-semester

Peer assessment

Peer assessment 09.00-10.30 Breakout room 1 - DOBLÓN

students. The performance consisted of two parts: (a) statistical knowledge, and (b) practical skills of analyzing a SPSS dataset. Prior to the direct-performance assessment, each student assessor answered a multiple-choice questionnaire to gauge their domain knowledge on statistical topics. Student assessors received a paper-pencil rating sheet consisting of checklists and global-ratings measuring junior students’ knowledge and skills. The rating sheet served also as a guide for providing feedback. Student assessors’ feedback statements were categorized according to their “Feedback Nature”, “Feedback Orientation”, “Feedback Function”, and “Feedback Elaboration”. Three student assessors were high in prior-knowledge and one scored was low. Most statements were cognitive rather than affective, and mostly task-oriented (process- and person-oriented state-

31

Thursday 28 august, 2014 ments were hardly observed). Although one student assessor was much lower in prior-knowledge, there was no difference in terms of cognitive or affective nature – however, the feedback was less task-oriented and contained fewer elaborate explanations. In

sum, a higher degree of prior domain knowledge appears to indicate more elaborated explanations aimed at the task at hand, but a more substantial sample is required for conclusive patterns.

Paper 3 Effects of peer feedback content and sender’s competence on perceptions and mindful cognitive processing of written peer feedback Markus Bolzer, Jan-Willem Strijbos & Frank Fischer

Feedback on student performance is central to formative assessment approaches. Various feedback types produce differential effects on feedback perception and performance. In peer feedback, students often perceive their peers’ competence to provide feedback as inadequate, resulting in low application. Feedback literature identifies mindful cognitive processing of (peer) feedback as important for its efficiency. A systematic investigation of such mindful cognitive processing together with the impact of feedback content and senders’ competence level is still lacking. In this study, the impact of peer feedback content and sender’s competence level on mindful cognitive processing has been investigated. A 2 x 2 factorial design varying feedback content (concise general feedback [CFG] vs. elaborated specific feedback [ESF]) and competence of the sender (high vs. low) was conducted with 45 psychology students. They received a scenario containing an essay by a fictional student, feedback by a fictional peer, a feedback perception

Peer assessment 09.00-10.30 Breakout room 1 - DOBLÓN

questionnaire, an essay revision task, a distraction task and a recall task. Eye tracking technology was applied as it enables the measurement of how the written peer feedback is read, e.g. which words, sentences, and remarks have been read or re-read in predefined areas of interest. Together with information on what has been revised, what has been recalled and how the peer feedback was perceived, conclusions about mindful cognitive processing can be drawn. Feedback from a high competent peer was perceived as more adequate than feedback from a low competent peer. ESF led to a more positive affect than CGF. No effects were found for competence level and/ or peer feedback content on revision performance or feedback recall. With respect to mindful cognitive processing, fixation durations were negatively correlated to revision performance in both conditions. Revision performance was positively correlated with feedback recall across conditions.

Test and assessment development Paper 1 The difficulty of item relevance in progress testing: the use of a rubric 09.00-10.30 to appraise item relevance Breakout room 2 - COMENDADOR Xandra Janssen-Brandt, Arno Muijtjens, Dominique Sluijsmans & Erik Driessen

To guarantee the validity of the progress test in midwifery education it is necessary for items to be relevant. The aim of this study was to research the applicability and feasibility of a rubric to appraise item relevance. This rubric is already used at Maastricht University, but has not been validated yet. From a PT 50 items were sampled. Students, teachers, members of the PT review committee (PTRC) and alumni participated as stakeholders. They were asked to rate for each of the 50 items the criteria of re-

32

levance as defined in the rubric, the overall relevance and to decide whether the item should be included in the PT. Finally the participants were asked to fill in a questionnaire about the feasibility of the rubric. It was shown that the criterion ‘medical knowledge’ was rated the highest, meaning that most items tested real medical knowledge. In general, overall relevance was rated higher than four of the five criteria of the rubric, indicating that the overall relevance is a compromise taking into account all five relevance

Thursday 28 august, 2014 criteria. The criteria ‘ready knowledge’ and ‘knowledge foundation’ seemed to be the greatest predictors in rating item relevance and the decision to include an item in the PT. The agreement about item relevance differed somewhat within and between the different groups. But besides these differences the decision to include an item was uniform. Items with a

low p-value were assessed as less relevant and not to be included in the PT. These less relevant items also scored high on the use of the ‘do not know’ option. In general the rubric is rated as a feasible instrument. Further research is focused on the validation of the rubric.

Test and assessment development Paper 2 Learning progress assessment in reading: How can we construct 09.00-10.30 and evaluate tests to monitor student reading progress? Breakout room 2 - COMENDADOR Natalie Förster & Elmar Souvignier

The assessment of learning progress supports teachers in keeping track of students’ performance development in class, enabling educators to adjust their teaching to individual needs. Curriculum-Based Measurement (CBM) is an established approach to assess learning progress by applying parallel forms of short tests throughout the school year. A prerequisite for the assessment of learning progress is the availability of short and technically adequate tests. The aim of this study was to examine the technical adequacy of eight newly developed computer-based tests for monitoring student reading progress following the CBM approach. In contrast to the commonly used reading aloud or maze task, this instrument is based on hierarchical models of text comprehen-

sion and assesses both basal reading skills as well as higher processes of reading comprehension. A total of eight parallel reading tests were developed. In a first study, 508 fourth-grade students completed these tests at intervals of three weeks over a period of six months. At the beginning and at the end of the study, students also finished two standardized reading tests to examine the construct validity of the CBM tests. Results indicate that the eight CBM tests show satisfactory reliability (.86 < Cronbachs α < .95; .70 Overall, computer-based assessment of students’ reading progress has proven to be applicable in general education. Initial analyses show a good psychometric quality of the new CBM reading tests.

Test and assessment development

Paper 3

09.00-10.30

Context in mathematics examination questions Jackie Greatorex

There is an ongoing debate about whether mathematics examination questions should be set in context. This literature review revisits the debate, focusing on public examinations in England. Initially taxonomies of context are abridged and an evaluation of the methods used to research the effect of context on candidates’ cognition, responses and performance is provided. The prevailing wisdom is that timed examinations should concentrate on the essence of the school subject and context is best used in projects undertaken with teachers’ guidance. However, context is sometimes required in timed written examinations.

Breakout room 2 - COMENDADOR

Therefore the advantages and disadvantages of examining in context are reviewed and research findings are used to make suggestions for developing questions with high quality contexts. An example is that familiar contexts are more accessible to learners and are therefore arguably of high quality. However, if the construct to be assessed is whether learners can apply mathematical principles and skills in new situations, then examining in familiar contexts is invalid. Consequently the suggestions for developing high quality contexts should only be followed if they do not compromise validity.

33

Thursday 28 august, 2014 Feedback

Paper 1 Receptivity to feedback: Research and implications for teacher/pupil interaction Jeffrey Smith, Anastasiya Lipnevich, David Berg & Marg Kendall-Smith

How do we receive feedback? Does the nature of our response to receiving feedback depend more upon the nature of the feedback and the context in which it was received, or more on us as individuals? Is receptivity to feedback more a trait or a state? The purpose of the research presented here is to extend our thinking about the effects of assessment for learning and the provision of feedback to students. We conduct an experiment to examine the question of whether the response to feedback is a more or less consistent characteristic of individuals (varying from one individual to another), or whether it is a phenomenon that varies according to the characteristics of the setting and nature of the feedback received. The experiment was conducting in college classrooms in the United States and New Zealand.

Mohd Nasri Awang Besar, Mohamad Nurman Yaman, Muhammad Arif Kamarudin, Tong Seng Fah, Saharuddin Ahmad & Maddalena Taras

34

Breakout room 3 - ESCUDO

Our results suggest that how people receive and react to feedback is more a trait of individuals than being dependent upon the context of the feedback and expectations of the individual receiving the feedback. We discuss the findings in terms of what they mean for teachers working with students in classroom settings. If some students are naturally more receptive to feedback than others, and if this individual disposition is more important than whether students are receiving feedback that they find pleasing or displeasing, what does this mean for teachers working with students? How can they develop in students a positive disposition toward receiving and using the feedback that is provided to them? We also look at a research program that will help to develop the ideas presented here.

Paper 2 Work Based Assessment: Effects of Feedback on Students’ Achievement in a series of Mini Clinical Evaluation Exercise (Mini-CEX)

Mini Clinical evaluation Exercise (Mini-CEX) is a work-based assessment that include feedback component that can lead learners to learn through their strengths and weaknesses of their clinical performance. This study was designed to analyse the learners’ achievement in a series of mini-CEX in family medicine posting that was conducted in a medical school in Malaysia. An analysis of 69 final year medical students who performed on a similar level of case complexity in two mini-CEXs was conducted. They were categorized based on their first mini-CEX scores, namely ‘poor’ (4.00-5.40), ’moderate’ (5.41-6.90) and ‘good’ (6.91-8.40). There were 35, 30 and 4 learners classified as poor, moderate and good respectively. The results showed that 55% learners have increased, 36% decreased and 9% sus-

09.00-10.30

Feedback 09.00-10.30 Breakout room 3 - ESCUDO

tained their scores in their second mini-CEX. There was a significant improvement in overall between first mini-CEX and second mini-CEX (mean different=0.30±0.99, p=0.016). Further analysis showed that there was significant improvement among poor learners (mean different=0.71±0.88, p=0.005). However, the good group showed a significant lower score in their second mini-CEX score (mean difference=0.87±0.25, p=0.006) but the score remain higher than the poor and moderate group. We concluded that feedback in mini-CEX played active role in improving the performances among poor learners. Further analyses needed to analyse the content of the feedback and its effect to the learners’ achievement.

Thursday 28 august, 2014 Feedback

Paper 3 Feedback giver: effects on feedback use and implementation Anna Espasa, Teresa Guasch & Montserrat Martinez-Melo

This paper aims to examine whether the feedback giver (teacher or teacher and peer) has different effects on writing assignments in an online learning environment. Research has traditionally highlighted the relevance of teacher feedback, but there is little research which analyses if students’ use and implementation of feedback is influenced by who gives it, whether teachers or peers. To achieve this aim, a quasi-experimental research was conducted (133 students, online Psychology Bachelor’s degree programme) during a collaborative writing assignment. In order to analyse students’ use and implementation of the feedback received, a methodological model was used, which is made up of three dimensions: student participation, the nature of student learning, and the quality of student learning. The nature of student learning is defined as the actions or activities that students do when receiving feedback (use of feedback). It includes cognitive, affective and metacognitive activities. The quality of student learning is understood

09.00-10.30 Breakout room 3 - ESCUDO

as the changes made by the students to produce the final version of the assignment (feedback implementation). Concerning the use of feedback, main results obtained show that when feedback is given by the teacher, there is a significantly higher percentage of cognitive activities. In relation to feedback implementation, whether feedback is given by the teacher, or teacher+peer, it does not impact on it (mark and improvement from draft to final text). However, results obtained in this research enable us to identify a specific situation where who gives the feedback makes a difference; namely when feedback generates affective activities in balance with metacognitive activities. In this case, students receive better marks and show greater improvement when it is the teacher who gives the feedback, especially of the epistemic and suggestive type. This paper contributes to facilitate making decisions on which kind of feedback to promote depending on the feedback giver.

Assessment literacy

Paper 1 Skilful compliance or critical stance? Assessment literacy in academic communities

11.00-12.30

Birgit den Outer & Margaret Price

Assessment practice in UK higher education institutions is substantially different from what it was a few decades ago. Assessment has become more complex and diverse, serves a number of purposes and involves various stakeholders. Increasingly, assessment takes place in an era of commodification of education where complex teaching and learning processes have been transformed into products that hold an exchange value, used for comparison and competition. To negotiate assessment both students and staff require a particular kind of competence; what it means to be a competent student has changed dramatically and, equally, what it means to be a competent teacher has changed too. In this paper, we investigate what this competence– we have


named this assessment literacy - might look like. An earlier conceptualisation outlines the technical and practical characteristics and offers recommendations for educational practice that can facilitate its development based within strong academic communities. Building on this, we look at assessment literacy as a socially situated practice by considering the experiences of 20 undergraduate and postgraduate students at a post-1992 university. Using audio diaries as a data collection tool and a social linguistic framework for analysis, the study puts under the lens the everyday world of students and their negotiation of assessment. We analyse the language students use about their negotiation of assessment using four categories: con-

35

Thursday 28 august, 2014 tent specific requirements, assessment techniques, general study and learning skills, and ‘playing the assessment game’. We conclude by considering two, not necessarily congruent domains within which assessment literacy can be considered a competence: first, assessment literacy as skillful compliance with

assessment in order to gain membership of an academic community. Second, assessment literacy as a competence that involves a critical positioning to, and ownership of, the learning process. The implications for the assessment literacy of teachers are explored.

Assessment literacy

Paper 2 Assessing the assessment literacy: The effects of a training programme on lecturers perceptions Victoria Quesada, Gregorio Rodríguez-Gómez & Marisol Ibarra-Sáiz.

As Price et al. (2012) argue lecturers must understand assessment if they are to develop students assessment literacy. The current study aims to analyse and evaluate the effect of a training programme based on assessment on lecturers’ perceived importance, competence and use of assessment. To do so, a pre-test/post-test quasi-experimental design with control and treatment groups was employed. A questionnaire was applied before and after the treatment to analyse lecturers’ perceptions of their assessment practices. In between, the lecturers of the treatment group participated in a training programme, re-designed one of their courses and implemented it. The sample consisted of 70 lecturers. A paired-samples T-test was conducted to analyse whether there were

Assessment literacy

Examining the Assessment Literacy of External Examiners Emma Medland

36


significant differences among each match paired group, also an independent-samples T-test was applied to analyse whether there were significant differences in the results of the treatment and control groups. The results show that there was a significant difference in the pre-test post-test scores in three of the four categories of the treatment group. No significant difference was found in the control group’s pretest and post-test results. Regarding the differences between the treatment and control groups in the post-test ratings, the scores differed significantly in three categories: ‘Assessment planning and design’, ‘Student participation in the assessment process’ and ‘Improvement and adjustment of the assessment practice’.

Paper 3

External scrutiny of HE courses is evident worldwide, but the use of external examiners from another institution is a distinguishing aspect of UK HE, and one that is highly valued. However, whilst guidelines ensure comparability / reliability of external examining procedures, the quality of the underlying practice does not inspire confidence. It is argued that both subject expertise and assessment literacy underpin the role of external examiners. But, this gives rise to two challenges. Firstly, if assessment practices are co-constructed and embedded within local institutional cultures and examiners are external to these, this will limit the extent to which they

11.00-12.30


can draw upon and apply their subject expertise. If this is taken to be true, the assessment literacy of examiners becomes far more pertinent. Fundamental to assessment literacy is a strong relationship between assessment theory and practice, but there is an abundance of evidence highlighting how weak this alignment is. This gives rise to the second challenge. There is an assumption that examiners are assessment literate but there is little, if any, evidence to support this. This paper addresses this gap in the literature by analysing all of the external examiner reports submitted to an institution during 2012-13. An intrinsic case study approach is adopted, and an

Thursday 28 august, 2014 open thematic qualitative analysis is used to analyse the reports. Price et al.’s (2012) characteristics of assessment literacy will then be used as a theoretical framework against which to evaluate the assessment literacy demonstrated. Initial findings suggest that the quality of the reports submitted is variable and

includes evidence of limited assessment literacy, thereby providing an example of the assessment theory-practice gap and a flaw in the external examining system that needs addressing if this gap is to be bridged.

Assessment literacy

Paper 4

11.00-12.30

Assessment literacy for mixed-age teaching Robbert Smit

Mixed-age or multigrade teaching, like it often occurs in small, rural schools but also for pedagogical reasons in some larger schools looks particularly interesting, when it comes to assessment for learning (Hargreaves, 2001; Little, 2007). In multigrade classes the teacher focuses more the individual student and applies differentiated learning. This is done based on formative assessments that supports children’s learning and documents their progress (Smit & Humpert, 2012; Stone, 1996). Effective assessment tools for mixed-age teaching are portfolio and children’s presentations in which they verbally or visually display their achievements to others (Guskey, 2003). Results from formative assessments help teachers to adapt their teaching and plan the next learning steps for the different grades or individual students. Multigrade teachers need to develop their assessment literacy in discussion with their team members as part of a professional learning community (Mulryan-Kyne, 2007). The main research question is whether a concept


of formative assessment within the group of multigrade teachers can be empirically demonstrated and which context variables are related to such a concept. As part of the project “schools in alpine regions 2”, we collected data from a voluntary sample of 271 teachers with multigrade classes in two regions in Eastern Switzerland and one in Western Austria. We administered a questionnaire in the beginning of 2013 and interviewed a smaller sampler of teachers from 30 schools in late 2013. With the help of a structural equation model it could be shown, that teachers with multigrade classes use a combination of formative assessment practices and tools in order to individualize and differentiate their teaching and instruction. More grades in class and higher self-concept of teacher’s competency to deal with heterogeneous classes relates positively to formative assessment literacy. Results from the analysis of interviews will follow and give more detailed insights.

Assessment in teacher education Paper 1 Should teacher education be paying more attention to student teachers’ 11.00-12.30 perceptions of formative assessment? Breakout room 1 - DOBLÓN Donna Hurford

How do we know what student teachers need to know about formative assessment to ensure they become effective classroom practitioners who value and integrate formative assessment into their daily practice? Is it a question of ‘developing the necessary assessment knowledge and skills’ as suggested in the introduction to this conference theme? Or should we listen more carefully to student teach-

ers’ experiences of assessment and in turn seek to develop courses which better address their needs? The presentation reviews research into the significance of environmental and personal factors which may affect student teacher and adult learner engagement with formative assessment. The focus of this study is on the initial analysis of phenomenographic interviews with newly qualified teachers who had

37

Thursday 28 august, 2014 recently completed the same post graduate course in teacher education at a UK university. The interview data provide insights into student teachers’ perceptions of formative assessment in school classrooms where they observed and practised teaching. The interviews reveal these emerging teachers’ critical self-awareness. The student teachers identify stages of readiness in their development as practitioners of formative assessment. Furthermore they reflect on factors which they perceived as enabling or constraining their development as practitioners of formative assessment in the classroom. The diversity of their perceptions of formative as-

sessment and the significance they attribute to different factors in their professional development reinforce the complexity of designing appropriate courses on assessment in teacher education. In conclusion, questions are posed about the relationship between a student teacher’s engagement with formative assessment as an adult learner and as a classroom practitioner: Is it important for student teachers to be self-regulated learners or can they forego engagement with formative assessment regarding their own learning and implement it effectively in the classroom?

Assessment in teacher education Paper 2 Action Research for the Promotion of Assessment Literacy 11.00-12.30 among Student Teachers Breakout room 1 - DOBLÓN Maureen Rajuan

This research focuses on the development of student teachers’ assessment literacy in the framework of an action research project. A working definition of teacher research is “”systematic, intentional inquiry by teachers about their own school and classroom work”” (Cochran-Smith and Lytle, 1993, p. 23). Action research empowers teachers to define problems they are having, design procedures for investigating solutions, implementing solutions in their own classrooms and assessing the results (Finch 2000). The research project was carried out by Israeli student teachers under the supervision of their teacher trainer (the researcher) in the framework of the English Department of an Israeli teacher training college in collaboration with the cooperating tea-

chers in PDS schools over a period of three years. Student teachers were exposed to a wide array of assessment tools and encouraged to design their own tools that best answered the questions that concerned them most. Qualitative results of questionnaires from 47 student teachers who participated in action research projects were collected. Preferred assessment tools designed by student teachers included peer feedback checklists, questionnaires to pupils and pupil interviews that were systematically used to collect data over time. It was found that the design of their own assessment tools greatly promoted personal reflection, a pupil-centered approach and innovation in the classroom.

Assessment in teacher education Paper 3 Undergraduate initial teacher education students’ peer assessment 11.00-12.30 and feedback communities: a longitudinal study Breakout room 1 - DOBLÓN Rita Headington

This study explored how three-year undergraduate primary initial teacher education (ITE) students developed and employed informal learning communities for the discussion of assessment and feedback. It examined the characteristics of these communities and whether they were used to interpret

38

and integrate the students’ experiences of assessment as ‘teachers’ and ‘students’. Using Gielen et al’s (2011) peer assessment inventory, it focused upon ‘composition of assessment groups’ and ‘interaction between peers’. Data were collected each year using questionnaires with one cohort of 100 ITE students,

Thursday 28 august, 2014 and audio diaries and interviews with 12 of these students across the second year of their degree. Social Network Analysis was utilised to identify informal learning communities used by students in the cohort at three points during their degree, to identify networks and interactions, and examine flow and influence (UKSNA, 2014). Visualisation and analysis of the SNA questionnaire data were achieved using Pajek 3* software. NVivo 9 was used to explore the diary and interview data, and determine the nature of interactions and quality of assessment literacy (Price et al, 2013) evident within students’ informal learning communities. Trust and continuity emerged as inter-related themes. Data suggested an ‘ebb and flow’ of students’ informal learning communities, based on trusting

relationships, shared goals, situational and emotional needs. Although continuity appeared to offer stronger levels of trust, the ability of individuals to provide appropriate and challenging feedback could accelerate or stultify development. Gielen, S., Dochy, F. and Onghena, P. (2011) ‘An inventory of peer assessment diversity’, Assessment & Evaluation in Higher Education, 36: 2, 137-155 Price, M., Rust, C., O’Donovan, B., Handley, K. with Bryant, R. (2013) Assessment Literacy, Oxford: Oxford Centre for Staff and Learning Development. United Kingdom Social Networks Association (UKSNA) (2014) What is Social Network Analysis?, available at http://uksna.com/sna.html, accessed 31 January 2014.

Assessment in teacher education

Paper 4

Assessing changes in teacher self-efficacy of preservice teachers over time Franziska Pfitzner-Eden

Teacher self-efficacy (TSE) is a powerful predictor of a range of positive outcomes for teachers and students alike. This research examined how TSE changes during teacher education at university. It relates changes back to two different types of inputs: theoretical (university course work) and practical (practicum at school). Participants are in two cohorts of preservice teachers, one at the beginning of a teacher preparation program in Germany, and one at an advanced stage of the same program. TSE was


assessed at the beginning and the end of the university term/year, and after the practicum at the start of the following term/year. Latent true change modeling was applied to analyze intra-individual changes over time. Results showed differential patterns of latent changes for the two cohorts, the two inputs, and the three different dimensions of TSE (instructional strategies, classroom management, and student engagement) examined.

Student characteristics, assessment and learning Paper 1 The Assessment of Underachievement in Secondary School Physics: 11.00-12.30 Insights into Achievement Patterns and Gender Differences Breakout room 2 - COMENDADOR Sarah Isabelle Hofer

The present study assessed physics underachievement by means of latent profile analysis (LPA) to identify intelligent students performing below their intellectual potential in physics. The sample consisted of 135 students from higher education in Switzerland (M=16.6; SD=1.37). Intellectual ability scores, physics grades, and physics conceptual understanding were used as profile indicator variables. A four

profile solution with gender as moderator proved to be best. Math grades and GPA (grade point average) were included as external outcomes to investigate how profiles differ on those variables. A problematic profile of female physics underachievers (14% of all girls; 61% of high ability girls) with high intellectual ability but below average physics grades was identified. Their math grades and GPA, by contrast,

39

Thursday 28 august, 2014 turned out to be within the normal range. They had a conceptual understanding showing existing albeit unexploited potential. Boys slightly underachieved both on a high and medium intellectual level in

Paper 2

terms of their grades while performing well on conceptual understanding. The boys’ underachievement was not limited to physics classes but seemed to be a general phenomenon.

Student characteristics, assessment and learning How gender influences performance assessment: 11.00-12.30 Student-teacher gender interaction in focus Breakout room 2 - COMENDADOR

Katarina Krkovic, Samuel Greiff, Sirkku Kupiainen, Mari-Pauliina Vainikainen, & Jarkko Hautamäki

The teacher assessment literacy does not only involve mastering the assessment procedures but also correctly making decisions in the assessment. Hence, teachers must be aware of all factors that impact their decisions. For instance, the gender stereotypes can highly impact the motivation of students to perform, but also influence the way teachers assess boys and girls (cf. Jones & Wheatley, 1990). Moreover, the teacher gender can play an important role, as well as the student-teacher gender interaction (Dee, 2006). In the present study, we aim to answer the question of whether and how the student-teacher gender interaction impacts teachers’ assessment of students, while we control for objective measures of students’ performance. The sample consisted of 1974 Finnish 6th grade students (Mage= 12.67) and their respective class teachers. Students completed mathematical thinking test, reading comprehension test, and scientific reasoning test. Furthermore, teachers provided assessment of each student, assessing students’ performance in different school subjects and regard-

ing their probability of academic success. In order to examine our research question, we applied multilevel analyses accounting for between- and within-class effects. Thereby, we controlled for the effect of students’ objective performance on teachers’ assessment and main effects of gender. The results indicated that the interaction between student and teacher gender did not influence teachers’ assessment in any subject. However, regardless of their gender and performance, teachers tended to assess girls as better than boys in first language performance (i.e., Finnish language) and potential in the overall evaluation for success in school after controlling for objective performance assessments. Teacher gender did not play any role. Such results suggest that the interaction between teacher and student gender is unlikely to be a source of possible bias in the assessment of students, however the effects of the student gender on the assessment should not be underestimated.

Student characteristics, assessment and learning Paper 3 Relationship between the English as a foreign language strategy use of 11.00-12.30 elementary school students and their general learning characteristics Breakout room 2 - COMENDADOR Anita Habók & Katalin Doró

Both researchers and language teachers have recognized the influence that learning strategy use may have on the acquisition and learning of a second or foreign language. It is now widely assumed that strategies used by successful language learners can be taught to beginners or less successful students, and, through this, facilitate their language learning. The development of good learning habits are also key for learners in be-

40

coming autonomous learners as classroom teaching can only provide some guidance in learning. This study wishes at contributing to this field by investigating a group of Hungarian EFL learner’s overall language learning strategies in grades 5 and 6 (n=311) and their general learning characteristics. The main research aim of this study was to investigate the relationship between students’ general learning characteristics and their foreign language learning

Thursday 28 august, 2014 strategy preferences. A background questionnaire informed us about students’ grades, attitudes towards their subjects and favorite English language activities. Two main questionnaires were employed: a) an adapted version of the SILL (Oxford, 1990) for learners of English as a second or foreign language (Doró & Habók, 2013) including 50 items on memory, cognitive, compensation, metacognitive, affecting and social strategies, and b) a learning characteristics questionnaire, including 49 statements in the following areas: effort and perseverance, instrumental motivation, learning strategies, self-esteem, students’ opinion about school and teachers, support,

cooperation, competition and critical thinking. When correlating the strategy questionnaire data with specific answers in the learning characteristics questionnaire, some correlations were found between some groups of strategies and specific statements concerning students’ the learning characteristics. An outcome of this research could be more direct research-based EFL teaching and the results should be used in pre-service and in-service teacher training. Results could also assist teachers in their curriculum design and in helping students develop autonomous English learning skills.

Student characteristics, assessment and learning Paper 4 Using large scale PIAAC assessment data to empower professional development: 11.00-12.30 Problem-solving in the context of technology-rich environments Breakout room 2 - COMENDADOR Raija Hamalainen, Bram De Wever, Antero Malin & Sebastiano Cincinnato

Currently, workers with vocational education and training (VET) background are facing a major technology challenge, as there is a snowballing need for continuous professional development to deal with the evolving character of technology (e.g. in the contexts of large industries). Thus, the needs of the current European workplace are challenging workers’ professional expertise with regard to problem-solving in technology-rich environments (TRE). The aim of this presentation is to utilize the OECD PIAAC (Programme for the International Assessment of Adult Competencies) data as a starting point for enabling and enhancing workplace learning and professional development. More specifically, we aim to use PIAAC data to provide indicators of the knowledge and skills of adult population and to shed light on a range of factors that contribute to thriving/successful adults’ skills. In short, PIAAC is a large-scale, ongoing programme for monitoring performance in literacy, numeracy, and technology-rich problem-solving

among adults. In this study, we will focus on the assessments of problem-solving in TRE in Finland. The data collection encompassed both the tests of problem-solving skills in TRE and the background questionnaire including questions about skills used. 5464 adults among 16-65-year-olds were involved (weighted response rate 66%, sample 8099) in Finland. 4503 adults participated in the test concerning problem-solving skills in TRE and answered the questionnaire about skills used. The overall analysis showed wide-ranging distribution of skill levels for adults. Additionally, our study aimed to find the background indicators for problem-solving skills in TRE. We found statistically significant indicators for workers’ skills that we will discuss in our presentation. Finally, we will illustrate the model of the educational sphere of influence that can be applied in developing new approaches to support workers’ professional development related to problem-solving in technology-rich work settings.

41

Thursday 28 august, 2014 E-assessment

Paper 1 Backwash Potentials: Reporting a Summative E-Assessment trial Rebecca Hamer, Antony Furlong & Sarah Manlove

The International Baccalaureate (IB) is developing an onscreen examination interface for its Middle Years Programme (MYP), ages 11 to 16 that will change the way students are assessed at this level. This paper presents the results of a full-scale trial of the first generation of online summative eAssessments. In total 71 IB Coordinators, 134 IB Teachers and 2,367 students completed detailed question-

Rebecca Eigenmann, Christin Siegfried, Juergen Seifried, Eveline Wuttke, Kristina Koegler & Marc Egloffstein

Attila Pásztor, Benő Csapó & Gyöngyvér Molnár

42

E-assessment 11.00-12.30 Breakout room 3 - ESCUDO

traits and states such as interest in the domain or subject, willingness for exertion or self-efficacy (Rychen & Salganik, 2003; Sembill, Rausch, Koegler, 2013; Sembill, 1992; Wuttke & Wolf, 2007). While the vast majority of corresponding research does not emphasize these aspects, our process-related approach explicitly integrates the modeling and measurement of non-cognitive facets of vocational competencies. A pilot study was conducted in May 2013 with N = 86 industrial clerks and N = 37 management assistants in IT-systems (reference sample) towards the end of their apprenticeship. The results illustrate the curricular validity of the problem scenarios being used and underline the importance of interest and self-efficacy during problem-solving processes.

Paper 3 Computer-based diagnostic assessment of thinking skills – the case of combinatorial reasoning Combinatorial reasoning plays a central role in scientific reasoning, problem solving and creativity. In everyday school practice, diagnostic tests should be applied to provide detailed information about the structure of the skills and the effects of differ-

Breakout room 3 - ESCUDO

naires at the end of the trial tests. A total of 296 student responses were submitted and marked by 11 examiners appointed by the IB. Of these, 113 were marked by both examiners and teachers. Results are discussed with regard to their potential to inform the development of positive backwash of summative e-assessments on teaching, curricula design as well on teacher’s digital and general assessment literacy.

Paper 2 Technology-based assessment of problem-solving competence in VET – results from a pilot study

In the field of vocational education and training (VET) there is a growing body of research on modeling and assessment of competencies. The German research initiative ASCOT (Technology-based Assessment of Skills and Competencies in VET; http:// www.ascot-vet.net) aims at developing test environments to assess vocational competencies in large scale designs. Within this research initiative, our project “Modeling and Measuring Problem-Solving Competence of Industrial Clerks” has been launched. In accordance with theories of problem solving and action regulation, one can assume that successful problem solving in any vocational context does not only require expert knowledge and general intellectual abilities, but also certain non-cognitive

11.00-12.30

E-assessment 11.00-12.30 Breakout room 3 - ESCUDO

ent content of the tasks or operations on students’ achievements. The administration of traditional paper-based and face-to-face tests is time consuming and because of the delayed feedback they cannot be used for diagnostic aims. Technology may provide

Thursday 28 august, 2014 solutions for these problems. The purpose of this study is to develop an online diagnostic assessment tool for the measurement of combinatorial reasoning and to analyse the effects of the different operations and task contents on students’ achievements. Participants were 216 fourth graders. We developed a computerized version of a paper-based test which measures different operations of combinatorial reasoning. The online data collection was carried out through the eDia (Electronic Diagnostic Assessment) platform via internet in schools’ ICT rooms. The reliability of the test (Cronbach α) was .88. Cartesian product was the easiest (M=53.1%), combination without repetition and finding all subsets were the most difficult operations (M=34%, M=32% respectively). Variation had average difficulty

Paper 1

(M=46%). The content of the items had significant effect on students’ achievements: items with letters and numbers were more difficult than items requiring manipulation of pictures (Mformal=35.43%; Mfigural=48.23%; t(215)=7.53, p