Modes of Professional Development

Modes of Professional Development An evaluation of the impact of different course modes operated across the National Network of Science Learning Centres

Judith Bennett Martin Braund Fred Lubben Yvonne Mason

March 2011

This report should be cited as: Bennett, J., Braund, M., Lubben, F., & Mason, Y. (2011). Modes of Professional Development: An evaluation of the impact of different course modes operated across the National Network of Science Learning Centres. Report commissioned by the National Science Learning Centre, York: University of York, Department of Education. The full data, including research reports from each Science Learning Centre and interview transcripts are stored at the Department of Education, University of York, and can be made available on request. © Copyright The University of York, UK, and the authors of the report hold the copyright for the text of the report. The authors give permission for users of the report to display and print the contents of the report for their own non-commercial use, providing that the materials are not modified, copyright and other proprietary notices contained in the materials are retained, and the source of the material is cited clearly following the citation details provided. Otherwise users are not permitted to duplicate, reproduce, re-publish, distribute or store material from the report without express written permission.

Modes of Professional Development

An evaluation of the impact of different course modes operated across the National Network of Science Learning Centres

Acknowledgements The research team would like to acknowledge the funding and support received from the National Science Learning Centre. The research team is very grateful to the following staff at the (regional) Science Learning Centres and the National Science Learning Centre who attended the research workshops and provided data and analysis for the study.

NSLC York: Andrea Mapplebeck, Irina Kudenko East of England SLC: Alison Redmore, Kate Mouncey East Midlands SLC: Sue Bull, Phil Hingley London SLC: Sally Johnson, Shirley Simon, Sheila Curtis North East SLC: Miranda Stephenson, Dr Carol Davenport, Mike Cole North West SLC: Amanda Smith, Dawn Jones South East SLC: Dr Janice Griffiths, Willeke Rietdijk South West SLC: Bryan Berry, Chris O’Callahan West Midlands SLC: Jo Flynn, Abbi Bryan Yorkshire and the Humber: John Wardle, Julie Jordan, Eleanor Brodie

We also wish to acknowledge staff at the schools involved in the evaluation, and other people who contributed in their various roles: Pam Handley, from the Institute for Effective Education at the University of York, who helped with training on using the Bristol Online Survey and advised on questionnaire design and analysis. Yvonne Mason, the administrator for the project, who tirelessly and efficiently dealt with communications and administrative matters and helped in production of this report.

TABLE OF CONTENTS Page

Executive Summary

iii

Section One

Background and context

1.1 1.2

Previous studies and evaluations Rationale and scope of the study

1 1 4

Section Two

Review of relevant literature

7

2.1.

Factors influencing classroom impact of PD for the systemic context Factors influencing classroom impact of PD for the school context

2.2.

Section Three The evaluation methodology 3.1 Aims and research questions

7 8 10 10 10

3.2

Methods

Section Four

Findings (1)

4.1. 4.2. 4.3

Selection and planning of professional development Impact of Modes of PD at Guskey Level 2 Suitability of Modes for specific Themes

16 16 21 26

Section Five

Findings (2)

27

5.1

5.4 5.5

What are the strengths and weaknesses of different modes of professional development in initiating and sustaining change in practice? Which particular modes of professional development are best suited to a particular type of action or change, such as development of leadership, curriculum development or subject pedagogy? What are the strengths and weaknesses of different modes of professional development in initiating and sustaining change in practice? The ‘routes model’ of CPD impact Schools’ implementation space

Section Six

Conclusions and recommendations

6.1 6.2 6.3.

General comments Choice of and expectations for CPD Initiating and sustaining change in practice (including dissemination of CPD outcomes) Evidencing impact Researching CPD impact Recommendations

5.2

5.3

6.4

6.5 6.6

i

27

35

39 43 46

48 48 48 50 51 51 53

Section Seven References

55

Section Eight

Annexes

57

Annexe A

Training schedules for the two research workshops at the NSLC Questions used in Bristol Online Survey instrument Interview schedules for case studies Case Study Analysis Framework for CELs

57 59 64 68

Annexe B Annexe C Annexe D

LIST OF TABLES Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Table 8 Table 9 Table 10 Table 11 Table 12 Table 13 Table 14

Guskey’s five levels of impact Cases in the evaluation Expected outcomes of CPD courses for the various Modes and Themes Self-reported factors influencing teachers’ choice of CPD Percentages of teachers per Mode considering factors an important influence on CPD choice Importance of the Mode of PD in selecting a CPD course by Mode Criteria for considering the Mode of PD important in the choice of CPD Teachers’ reported challenge of the CPD to their way of teaching Reported dissemination activities after CPD Beneficiaries of the CPD apart from the participants and their own pupils Teachers’ reported immediacy of usage of CPD generated teaching materials Teachers’ reported changes in their classroom teaching Impact of CPD Mode and Theme on pupils’ abilities Evidence of change in pupils’ skills, understanding of science concepts and attitudes

2 15 17 18 19 20 20 21 22 23 24 24 25 26

LIST OF FIGURES Figure 1

Classroom impact routes resulting from CPD input

ii

44

Executive Summary 1.0

Background and context

1.1

The report presents the findings of a study of the impact of different modes of continuing professional development (CPD) offered by the National Network of Science Learning Centres (NNSLCs). The report was commissioned by the NNSLC’s Research Impact and Accreditation Group (RIAG). The principal types of course offered by the NNSLCs fall into three broad groups: pedagogy, curriculum and leadership. The modes of professional development evaluated in the study are one-day sessions; block sessions of more than one day and up to three days (of which some were residential); multi-contact sequences with between session tasks; bespoke in-school delivery of a course. The study addressed the following research questions, which were set by the Research Impact and Accreditation Group (RIAG) - What are the strengths and weaknesses of different modes of professional development in initiating and sustaining change in practice? - Which particular modes of professional development are best suited to a particular type of action or change, such as development of leadership, curriculum development, subject pedagogy? - How do schools approach their planning and professional development given the choices available? The study also had as one of its aims the strengthening of the expertise of NNSLC staff in undertaking evaluations.

1.2 1.3 1.4

1.5

1.6

2.0

Methods

2.1

The study was undertaken by a team in the Department of Education at the University of York and a Centre Evaluation Leader from each centre of the NNSLCs. Centre Evaluation leaders attended two separate days of training, organised by the core team at the University of York. Data on different modes of CPD were gathered through an on-line survey of participants in selected courses (which used the Bristol On-line Survey package), and through detailed case studies that involved participants and their managers from one course provided at each of the NNSLCs. 150 questionnaire responses were submitted, and fourteen case studies were carried out.

2.2 2.3

2.4

iii

3.0

Key findings: choice of CPD

3.1

Choice of CPD courses is influenced by a complex set of interacting factors: cost, location, duration, likely disruption to teaching, disruption to personal lives and the personal expectations of teachers and their managers. Cost was a particularly high priority, and the impact awards crucial in allowing participation. Almost all questionnaire respondents described their CPD needs as ‘self-identified’. Impact was greatest when there was a close match between personal needs and institutional priorities. Multi-contact courses were selected most frequently because they were perceived as offering a thorough learning experience. The potential advantages of ‘bespoke’ in-school courses was recognised, but often outweighed by a desire to engage in CPD away from the business and distraction of the school environment. Half-day and twilight courses were not seen as particularly effective by teachers due to their short nature and teacher fatigue at the end of the day.

3.2

3.3 3.4

3.5

4.0

Key findings: expectations of CPD

4.1

4.2

Three-quarters of teachers expected their CPD to generate new teaching resources and strategies, and around one-third hoped to improve their knowledge. Half of the teachers expected to have a role in disseminating the outcomes of their course in their own schools. Participants in sequential courses more often expected to focus on networking or resolving policy/practice issues, while those involved in bespoke courses more usually expected to gain exemplary teaching materials and strategies for dissemination to peers and develop teacher networks.

5.0

Key findings: evidence of impact

5.1

Two-thirds of teachers indicated that their thinking had been challenged through engaging in the CPD, and this was the case for almost all participants in the leadership courses. Almost all participants self-reported impact on their own teaching but corroborating, ‘hard’ evidence of impact, for example from peer observations of teaching, pupil performance or attitude data, was rare. The most extensive evidence of impact on teaching and pupils was reported in cases of leadership courses using multi-contact CPD. However, it should be noted that the leadership courses had more specific expectations for the gathering of evidence. It was not possible to establish clear and direct links between any one mode of CPD and the three main types of impact (personal impact, peer impact, whole-school impact). Personal impact occurred even when the focus of courses was clearly on leadership to influence peers and the whole school. Impact is most secure where personal and departmental priorities for CPD harmonise with schools’ priorities for development or change.

5.2

5.3

5.4 5.5 5.6

iv

6.0

Recommendations to the NNSLCs for the design and delivery of CPD

6.1

The NNSLCs should consider the extent to which it is able to meet an increasing desire for bespoke and off-site whole school delivery of CPD. The NNSLCs should note the greater impact of multi-contact courses and leadership courses, and should consider how it can increase provision here, particularly across all SLCs. The NNSLCs should consider using the outcomes of this and other studies to make clear to potential ‘clients’ the values and advantages of different modes of CPD. The NNSLCs should consider clarifying how content and outcomes can be best matched to individual client’s requirements. The NNSLCs should be particularly sensitive to teachers’ perceptions of workloads and fatigue associated with half-day and twilight courses. The NNSLCs should ensure that action planning and target setting processes, such as ‘the toolkit’, associated with CPD are made clear and allocated sufficient time in all courses, especially those having single-contact. The NNSLCs should consider setting up systems to maintain closer contact with teachers who have attended single event CPD and encouraging them to provide evidence of impact of CPD, particularly for one-day or two-three sequential day courses. The NNSLCs should recognise explicitly that most teachers completing CPD wish to influence colleagues, and provide specific guidance and advice on all courses as to how this might best be achieved. The NNSLCs should consider taking account of the remote locations of some centres from SLCs and the possibility of strengthening local ‘hub’ or ‘bespoke’ provision.

6.2 6.3 6.4 6.5 6.6

6.7

6.8

6.9

7.0

Recommendations to the NNSLCs for the evaluation of CPD impact

7.1

The NNSLCs should consider making use of the online survey facilities of BOS or similar systems (such as ‘survey monkey’) to collect large scale data from courses provided across their networks, and the importance of collecting and sharing these data should be stressed to all participants in all courses. The NNSLCs should consider exemplifying examples and methods through which CPD impact on teaching and on pupils’ performance, attitudes and behaviour can be collected, analysed and communicated. The NNSLCs should consider reporting data on CPD/course impact via local communications; this is already done nationally by the NSLC on behalf of the NNSLC but SLCs have an opportunity to do this more locally, where case studies of successful impacts could be disseminated.

7.2

7.3

8.0

Recommendations for schools and colleges

8.1

Schools and colleges should use their existing monitoring and lesson observation systems more effectively to provide more robust and accountable evidence for classroom impact of CPD; this could include a requirement for teachers and departments to account for evidence of impact.

v

8.2

8.3

Teachers should be encouraged to collect examples of evidence of impact of CPD; this might include surveys of staff use of resources, questionnaire data on pupils’ attitudes to new teaching or evidence of improved pupil performance in tests, examinations and homework. In the case of bespoke courses, schools should ensure that they enter into a clear dialogue with SLCs on requirements and adaptations of courses to meet the school or college’s needs.

vi

1.

Background and context

The National Network of Science Learning Centres (NNSLCs) was established some six years ago through Government and charitable funding to provide continuing professional development (CPD) at its national and regional centres. The CPD provision is targeted at supporting science teachers in gaining knowledge, skills and confidence needed to change their practice in a number of key areas. These include science-specific areas such as the use of contemporary science in teaching; implementing new initiatives in the science curriculum, teaching, learning and assessment; practical work and ICT; and more general areas such as leadership and management of change. The aspiration for the programmes is to embed CPD in teachers’ professional practice such that it impacts on teachers’ attitudes to professional development, on classroom practice, on pupils’ experiences in learning science and on pupils’ attitudes towards school science and science more widely. The NNSLCs is jointly funded by grants from the Wellcome Trust and the Department for Education (previously known as the Department for Children, Schools and Families [DCSF]) and comprises nine Science Learning Centres (SLCs) serving regions of England and a National Science Learning Centre (NSLC) at York. The nine regional centres (SLCs) are mostly based in Institutions of Higher Education and offer courses for science teachers, school technicians and others of short duration (up to three days, but most commonly one or two days). The NSLC was provided with residential facilities and can therefore offer longer courses than at the SLCs. Two-part courses are common at the NSLC and comprise blocks of two or three days separated by weeks or months back at school. There are a few courses that involve three or four blocks of training, more often aimed at existing or potential managers or coordinators of science in primary and secondary schools. The other distinct characteristic of the NSLC is that it can provide courses for the devolved nations of the UK, whereas the regional centres are for England only. Between 2005 and June 2009, the NNSLCs delivered over 70,000 training days to primary and secondary teachers and school laboratory technicians. It is estimated that in 2008/09, the NNSLCs engaged in one way or another with 72% of all maintained secondary schools in England.

1.1.

Previous studies and evaluations

The network has developed a system of gathering feedback and evaluation on its programmes, which has largely drawn on self-reported data from participants. This includes the completion of a ‘learning and evaluation tool’ in which key learning is noted, together with an action plan, followed up some months after the course completion by a form gathering information on the impact of the professional development. The NNSLCs systematically collects evidence of course impact using Guskey’s (2002) impact model. Briefly, this model differentiates between the levels of impact of CPD summarised in Table 1.

1

Table 1: Guskey’s five levels of impact Level

Impact

1

Participant’s reactions to the course provision

2

Participant’s learning

3

Organisational (i.e. school or department) support and change

4

Participant’s use of new knowledge and skills

5

Student learning outcomes

Since the inception of the NNSLCs there have been a number of research studies of CPD impact carried out by the NNSLCs itself or as independent, commissioned studies. The NSLC produces Annual Impact Reports summarising the findings of these studies, headline data on participation in and content spread of courses and examples of evaluations of individual courses at regional level (see for example, NSLC, 2010a). A large scale evaluation of the NNSLCs was commissioned by the Department for Children, Schools and Families and the Wellcome Trust and took place between May 2006 and December 2007 (Wellcome Trust, 2008). The research was carried out a by a team at GHK Consulting and involved two sets of interviews with key informants involved in commissioning the NNSLCs and in science CPD more widely, telephone surveys of 1,400 course users and 100 individuals who had not accessed courses and observation and follow-up interviews with participants attending 22 courses at SLCs. The research team found that educators working for Government initiatives such as the National Strategies were initially concerned that the out-of-school model of CPD provision adopted by the NNSLCs was not the most effective one. This concern had abated, to a certain extent, by the end of the study period as SLCs were seen to provide more outreach courses. Impacts of courses on individual teachers were reported in terms of improvements in personal motivation, subject knowledge and confidence in the classroom. The authors of the report suggested impact took place in two stages. The first stage is impact at a personal level and is rarely sustained while at a second stage there are changes in pedagogy, management and leadership that are sustained over a longer period of time. The authors concluded that courses with a longitudinal component were more likely than single day courses to bring about changes at the second level. Wider impacts on peers or on whole schools were reported less frequently than personal changes. Over two thirds of responding teachers claimed there had been ‘some’ impact on their schools and about half claimed that course outcomes had influenced their colleagues. The research team identified a series of factors that they saw influencing positive impacts from SLC provision: 

A strong longitudinal component to provision – either through multiple or linked days of provision, or a wrap-round model featuring pre- and post-course tasks;



The identification of expected learning outcomes in advance of attending provision – and acting on this information to tailor provision as practicable; 2



The use of pre- and post-course activities – and the completion of any preparatory work prior to course attendance;



Post-course follow-up by the Centres.

The Wellcome study showed that SLCs produced mainly anecdotal evidence of impacts and, where evidence of impact studies had taken place, these were of one or two isolated examples of courses. One of the report’s key recommendations was for the NNSLC to establish a research and analysis team to increase capacity for research and provide research intelligence across the network (page x). This recommendation resulted in the formation of the Research Impact and Accreditation Group (RIAG) that commissioned the study reported here. In 2009, Mary Ratcliffe, at the NSLC, and her colleagues Alison Redmore (East of England SLC) and Catherine Aldridge (then of South West SLC) analysed impact forms returned by course participants to look for evidence of impact at the higher levels of Guskey’s scale (3-5) (Finegold, 2010, p. 13). They also wanted to know if the system of self reporting embodied in the NNSLC’s impact toolkit process was a reliable means of measuring impact. As in the Wellcome study, they showed that teachers most frequently reported gains in skills and use of new teaching methods and reported sharing new knowledge and understanding with their colleagues. Ratcliffe, Redmore and Aldridge were confident there were sufficient chains of evidence from documentation to support gains in participants’ professional knowledge and attitudes but less sure of the quality of evidence that teachers had collected and drawn on to evidence impact at classroom level. Two independent studies were commissioned by the NNSLCs in 2009. One was the study carried out by the authors of this report to explore, using a case studies approach, the impact of courses provided by SLCs on teachers’ skills, knowledge and confidence in relation to contemporary topics in science and consequently on their classroom practices. The second study by Scott, Ametller and Andrews at the University of Leeds probed impact of selected courses on teachers’ understanding of specific scientific concepts and associated changes in their PCK and their pupils’ understanding of science (Scott, Ametller and Andrews, 2010). Using a pre- and post-test either side of a two-part CPD in Physics, they found evidence of sustained enhanced learning for a disparate group of 15 teachers and evidence of substantial improvements in pupils’ learning of Newtonian mechanics. Bennett, Braund and Lubben, from the Department of Education at the University of York, studied 8 cases of teachers attending four courses at two SLCs and the NSLC. An important outcome of this study was to propose a new model accounting for and understanding change in practice and impact through CPD. Rather than seeing progress in CPD outcomes as purely hierarchical in terms of Guskey’s levels, they proposed a model showing gains and impacts on three ‘routes’ concerned with; personal professional gains (P1 route), impacts on peers (P2 route) and influencing or changing departmental or school policy (P3 route). Their study showed gains are possible simultaneously on all routes, with P1 gains being most common. In primary schools there was more likelihood that teachers on courses would bring back knowledge and skills that were quickly translated as P3 gains impacting school policy. The York team also described factors operating in what they called schools’ ‘implementation space for CPD’. Significant changes in practice characterised in the implementation space were often due to favourable combinations of schools’ structures for managing, monitoring 3

and supporting CPD, the personal positive characteristics and levels of motivation of participants and the relevance of course content to institutional drivers for change at department or school level. There was particularly strong impact on all routes where CPD had resolved disputed areas of policy or practice, for example, the definition of who are gifted and talented pupils and what chemistry practicals can be conducted under health and safety rules. Weakest change was in cases where approaches in pedagogy or curriculum suggested by courses clashed with departmental or school priorities or where schools had not coordinated sufficient follow up from the CPD.

1.2.

Rationale and scope of the study

The study was established with reference to a remit for research produced by the Research Impact and Accreditation Group (RIAG) of the NNSLCs in their invitation to tender. Thus research questions and suggestions for methods of the study were already set (see section 2). The study is a natural development from those already described, in that these previous studies considered impact without a systematic attempt to take account of the different styles, situation and durations of CPD contacts. Thus this study is concerned with impact due to different styles of organisation and delivery of courses. In this report ways in which courses are organised and delivered are referred to as ‘modes’. The term ‘mode’ rather than ‘model’ was chosen to avoid confusion with theoretical models devised to represent and understand CPD actions and outcomes (such as Guskey’s model and the ‘Routes’ model devised by the York team). In the invitation to tender, types of course modes of CPD operated across the NNSLCs were identified by the RIAG as follows: A. One day interactive CPD sessions at SLCs or other ‘off-site’ facilities. B. Residential or non-residential CPD, more than one day and up to 3-day sessions; C. Sequences of one day provision with an expectation of some tasks in between. D. In-school delivery of a core course where participants have a clear common background. E. In-school delivery of bespoke sessions identified collaboratively between school and course leaders. F. Action research. G. One day conferences. H. Course modes like the Science Additional Specialism Programme (SASP), with a dedicated cohort undertaking sessions and support across a year. Courses across the network are provided according to common professional development needs identified for all centres, known as ‘themes’ and communicated to teachers in brochures and webbased information. It seemed sensible to include exploration of impact of different modes of CPD provision and duration for some of these themes. 4

The common themes identified by the NNSLCs were: 1. 2. 3. 4. 5. 6. 7. 8. 9.

Contemporary science and how it works Leadership New initiatives in the curriculum Personalising the learning in science Practical work and ICT focus Science for non-specialists STEM curriculum innovation and implementation Supporting science teaching Teaching learning and assessment (NSLC, 2010b, pages 8-9)

Given the limited resources and time available for the project, it was decided and agreed by the NSLC and RIAG that it would not be feasible to research all modes and themes. It was agreed to concentrate on modes and themes most relevant for classroom teachers and leaders and in areas not subject to research or evaluation elsewhere. Thus courses on science for non-specialists (Mode H and Theme 6), action research (Mode F) and supporting science teaching (Theme 8) were excluded from the study. It was also decided to exclude conferences (Mode G) since these are often events organised by organisations outside the network and likely to have very low impact. In discussions with the NSLC, it appeared that Modes D and E share common ground in that they are both concerned with client specific delivery of inset at the client base. This mode was redefined as ‘bespoke’ (Mode D, as below). Thus the modes considered by the study were rationalised to these four:

A. One day (or single contact) interactive CPD sessions at SLCs or other ‘off-site’ facilities. B. Residential or non residential CPD of more than one day, for example some as 3 day sessions; C. Sequences of varying length provision (multi-contact) with an expectation of some tasks in between. D. Bespoke, in-school delivery of a core or unique course where participants have a clear common background. Courses were allocated to three super-ordinate themes: Curriculum (e.g. NNSLCs themes: 1, 3, 7), Pedagogy (e.g. NNSLC themes: 4, 5, 9) and Leadership (NNSLC theme 2). A definition of these themes is as follows:

5

Pedagogy The course/CPD has most, or at least more than half, of its contact time concerned with methods of how to teach a topic, concept or skill area or how to adapt teaching for a specific target group (such as ‘Gifted and Talented’ pupils). CPD that is concerned with how to communicate understanding of science skills and processes or concepts will be part of this theme as communication is often concerned with translation of science into teachable experiences for pupils and this lies at the heart of the pedagogy of science. Leadership The course/CPD has most, or at least more than half, of its contact time connected with how individuals might lead change and manage personnel or resources. Such CPD implies that participants will have or might want to have responsibility for a curriculum area such as science, science coordination (in a primary school) or a discipline such as physics, biology or chemistry. Thus aspiring leadership as well as existing leadership is covered by this CPD content theme. Curriculum This theme is harder to define as there are overlaps with what might be construed as pedagogy. A curriculum CPD will have much pedagogical content BUT the remit and rationale for the course probably stem from national or regional imperatives to develop or change the curriculum or introduce specific approaches or content. Courses addressing the introduction of teaching separate sciences, coordinating STEM teaching or cross curricular links between science and other non-STEM subjects would come under this theme. Where there is significant change to the curriculum or its assessment envisaged or already enacted by government or agency policy, courses that address this could also be included under this theme.

6

2. 0

Review of relevant literature

The review of the literature draws on and extends the systematic review of the literature on the nature and extent of change in classroom practice as a result of PD interventions in our previous report (Bennett, Braund and Lubben, 2010). This brief review focuses on classroom impact of PD interventions. As mentioned earlier, Guskey (2002) differentiates between five levels of impact of an intervention (see Table 1 page 2). At the most basic level, the PD intervention may affect the participant’s experiences. At the second level the PD may impact the participant’s knowledge skills or attitudes, including confidence. At level 3, the school organisation may be changed as a result of the intervention. Level 4, reflects the impact on the participant’s actions in the classroom, and level 5 reports the impact in the form of changes in pupils’ knowledge skills or attitude. The current study focuses on factors influencing Level 4 impact, and assumes, with Guskey, that any level 5 impact will indicate an appropriate level 4 impact.

2.1.

Factors influencing classroom impact of PD for the systemic context

Comprehensive impact studies (especially, Desimone et al., 2002; Garet et al., 2001) across a variety of PD projects, within the Eisenhower programme in the USA, identified nine characteristics of such PD projects influencing classroom change. They concluded that high-impact programmes focus on PD content with externally-imposed objectives (e.g. assessment standards) and deal with pedagogic content knowledge (PCK) of new curriculum aspects. The impact is also high if the PD delivery method uses a transparent planning process, formative evaluative cycles, allows for active learning of participants, is implemented in close collaboration with local education authorities, and covers an extended period of time. In terms of the PD context, classroom impact increases if the PD experience addresses teachers’ goals, and links closely with the reality of school. A subsequent study by Penuel et al. (2007) addressed the critique that the Eisenhower studies focused on the PD provision, and less on the implementation context (cluster, school, department, class) within which the PD experience needs to be embedded. Penuel and colleagues built on the framework constructed by Desimone et al. and found that classroom impact increases when the PD content emphasises subject content and PCK (not necessarily ‘new’ topics) or the teaching of inquiry methods. The impact is influenced by the total number of PD hours. A major factor in PD impact is the ‘proximity to practice’ (p. 928) with an increased impact where the PD experience allows for planning for integration in own practice; provides or exploits on-going, in-school support coherent with school practice and provides detailed resources supporting the innovation which should be closely related to the school context. In terms of PD programme duration, Darling Hammond et al. (2009) established that only programmes with more than 30 hours contact time over an extended period of several months show statistically significant positive outcomes on student learning (level 5 impact, according to Guskey, 2002). The impact on student learning supposedly was affected through positive classroom change. The American studies reported above provide a useful list of pointers for the nature of PD provision that will increase classroom impact. However, they may not be very informative for the current study, since the PD programmes being evaluated in these studies all supported centrally initiated 7

major curriculum innovations, which is not the main focus of the PD support provided by the NNSLCs. Loxley et al. (2007) differentiate between various types of contexts influencing PD impact, i.e. the teacher context, school context and systemic context. Thus, in Loxley’s terms, for the largescale studies above, the systemic context is the driving PD force and the school context is the implementation setting. In contrast the NNSLCs work in an environment where the teacher context and school context are the driving PD forces, and the systemic context forms the backdrop.

2.2.

Factors influencing classroom impact of PD for the school context

Describing the forces driving the professional development of British teachers, Day and Gu (2007) identify teachers’ need for variation and renewal during their professional life in order to remain an effective teacher, and they see PD as one of ways of fostering such teacher resilience. Based on a 4year longitudinal study using data from more than 300 primary and secondary teachers from a stratified sample of schools in England, Sammons et al. (2007) conclude that strong teacher resilience is expressed, amongst others, as change in classroom activities. They suggest that PD experiences will strengthen classroom impact if the PD experience takes account of personal, workplace and external factors. More specifically, Boyle et al. (2005) explored the relationship between the characteristics of the PD provision and the impact on classroom practice. More than three quarters of the sample of 800 primary and secondary teachers across England, participating in long-term professional development activities, reported they made changes in their lesson planning, assessment approaches and/or teaching style. Very few teachers (less than 10%) considered any of the modes of PD ineffective, and for all modes, including workshops, coaching and ‘sharing practice’, around half of those involved felt them to be very effective (Boyle et al., 2004). Calls for further research were made into the specific factors, or combination of factors, that make any of the PD modes being perceived as either reasonable or very effective. In a systematic review of the literature, Cordingley et al. (2007) surveyed the nature of the PD support provided by specialists (from inside and outside the school) for interventions that resulted in classroom change. The review included interventions with a duration of more than 12 weeks only. Common factors of the interventions included formal and regular input of the specialists over a period of several months, and the exploitation of in-school support by peers. PD with classroom impact had a large component of participant autonomy: specialists required participants to explicate ‘individual starting points’, allowed them structured time to develop bespoke materials and strategies which would be supported when applied to their own practice. Both the key studies by Boyle et al. (2004) and Cordingley et al. (2007) have limited themselves to identifying factors determining classroom impact for sustained PD interventions, probably because of the limited number of permutations of, and larger investments in, these long-term interventions. Although little research has been undertaken on PD support through short, specific, and targeted interventions, received wisdom assumes ‘more input, more impact’ (see also, Adey, 2004). An illustration of a small study of the classroom impact of a short CPD workshop to support the teaching of Earth science at KS3 is provided by Lydon and King (2009). They analysed questionnaire responses for classroom impact plus baseline data for the state of pre-workshop teaching of Earth 8

science in 15 participating schools. After one year all schools reported long-term changes, to various degrees, as evident from changed schemes of work to accommodate practical activities promoted at the workshop. Half of the schools reported considerable impact, adopting most or all of the proposed activities. The reported impact resulting from one 90 minute workshop contradicts earlier reported research, suggesting that CPD needs to be sustained for impact to occur. Lydon and King (2009) point out characteristics of the successful CPD intervention; it was grounded in an explicit theory of learning and promoted tried-and-tested teaching methods supported by high quality and detailed materials for a newly constituted area in the curriculum. Also teachers, who attended as whole science departments in the school-based CPD workshop, were encouraged to work in groups and share experiences and so to take ownership of the changes. These findings suggest that appropriate CPD content and a facilitating school context can offset the absence of desirable features of the CPD delivery strategy, such as an extended time period. The current study fills some gaps in research in that it focuses on factors influencing classroom impact of relatively short PD interventions, that is from 1 day workshops (mode A), several day workshops (mode B) to at least two blocks of several days input with an application period in between (mode C). Usually, each of these PD interventions was preceded by explication of ‘individual starting points’ and followed by structuring an implementation plan. Thus we address the relationship of three factors identified in the literature above: the duration and opportunities to plan for and practice change (comparing between the various modes) and the content of the PD (comparing between the various themes).

9

3.0

The Evaluation Methodology

3.1

Aims and research questions

Study aims were as follows: 1. 2. 3. 4.

to explore in more detail the nature of the impact of the CPD activities of the NNSLCs; to enhance the abilities of the member centres within the NNSLCs to collect and reflect on data relating to their own practice; to provide a means by which individual member centres within the NNSLCs can share practice in relation to the evaluation of their activities; to refine and develop the emerging Routes to impact model of CPD to provide a detailed theoretical framework for the impact of CPD provision.

The specific research questions (set by the RIAG in their invitation to tender) were as follows: 1. 2.

3.

3.2.

What are the strengths and weaknesses of different modes of professional development in initiating and sustaining change in practice? Which particular modes of professional development are best suited to a particular type of action or change, such as development of leadership, curriculum development, subject pedagogy? How do schools approach their planning and professional development given the choices available?

Methods

In line with the RIAG’s wish to improve capacity for and awareness of evaluation across the network, the study was designed to involve each Regional Centre (SLC) and the National Centre (NSLC) in data collection and analysis. The central research team at York asked each centre to nominate a person to be responsible for the operation of the project at their centre. These persons were called Centre Evaluation Leaders (CELs). CELs attended two day-long research workshops at the NSLC at which the aims of research, sample selection and data collection and analysis methods were shared and refined. Thus the aim to build research capacity across the network was met through development and deployment of centre-based expertise. Details of the research workshops are found later in this section and the programmes for each are appended (see Annexe A). Data on impact of the different modes of CPD were collected through an online survey of participants for selected courses and through detailed case studies involving interviewing participants and their managers for one course that had been provided by each of the centres.

3.2.1.

The selection of courses

The selection and allocation of courses for data collection using the online survey and from case studies was made in these stages. Stage 1

All core courses, that is those run across the network and delivered at each SLC, and courses at the NSLC were identified from course brochures and documents. Only 10

courses that appeared in the time frame September 2009 to April 2010 were selected. This time frame was set so that course outcomes might have had time to bed in and have impact in schools (about half an academic year) before the data collection and case study period (November 2010 to January 2011). Stage 2

Courses were allocated one of the four modes and one of the three super-ordinate themes (Curriculum, Pedagogy and Leadership) and only selected for study where there appeared to be sufficient numbers (more than 10) attending on each occasion courses were delivered.

Stage 3

The selection of courses from stage 2 was sent to each centre for authentication – that the course actually ran, that it attracted the number of participants stated in databases, that it had been delivered on more than one occasion and that the mode and theme identified by the central team were valid.

Stage 4

From this validated list, the central research team at York selected courses that could be evaluated at each centre according to a balance for mode and theme. These were given to CELs at the first training workshop.

Stage 5

Following discussions and readjustments at the first research workshop in September, CELs agreed to select one core course and were asked to add one other course for evaluation that had been delivered as bespoke training. Thus the intention was for each centre to evaluate two courses; one core and one bespoke.

Stage 6

At the second research workshop in November some CELs had changed their opinions as to the mode of core and bespoke courses agreed at stage 5, even though these had been through a validation process. There was also some discussion and doubts expressed about the capacity of CELs to collect data for two courses, bearing in mind that it was also agreed that two participants (schools) should be included as the case studies for each course.

3.2.2.

The on-line survey of course participants

The views and experiences of course participants were collected through a questionnaire using the electronic Bristol On-line Survey (BOS) package for easy access of respondents. The questionnaire consisted of twenty closed questions with single or multiple options and open comment boxes. Items focused not only on the selection process of the CPD course and the perceived impact of the CPD experience across Guskey’s levels, but also on the perceived reasons and available evidence supporting these views. The set of items was constructed and peer-validated through discussion between four researchers. Subsequent validation took place through an online pilot with an invited set of course participants outside the target group, and through vetting by the CELs during the first workshop (see below). Consequently, some questionnaire layout was changed, terminology was modified, items were reworded for greater clarity, and several option responses were added or removed. The sampling of course participants was purposive (see section 3.2.1 above). In short, the sample population consisted of all participants in a selection of CPD courses stratified for the four CPD 11

modes (A-D) and three content themes (Pedagogy, Curriculum and Leadership). A total of twenty courses were included, many offered at a number of SLCs. CELs invited participants in those courses run at their SLC and across the network, totaling around 1800 participants. Over a period of six weeks, 156 responses were received, six of which had no clear indication of a course code, and thus could not be classified for mode or theme. The remaining 150 responses formed the basis of the analysis. The stratified sampling of courses resulted in a reasonably balanced set of 150 responses across modes and themes. However, after data collection was completed, the mode allocation of courses had to be adjusted since some CELs had changed their opinions about the mode of the courses delivered at their SLCs. As a result, within the sample there is now little independence between mode and theme. The responses were analysed using simple statistical methods of frequency and percentage comparison between modes and themes, thus identifying patterns of differences. The reliability of the interpretation of the analysis was safeguarded since two researchers independently viewed the findings, and the responses for several items were discussed at the CELs’ research workshop 2.

3.2.3.

The research training

An important aim of the project was to develop a research capability and capacity at each centre so that future evaluations of courses might be more robust and research intelligence generated and shared across the network. To do this, and specifically train CELs to collect and analyse data for this study, the York team delivered two day-long research workshops at the NSLC, one in September 2010 and another, nine weeks later, in November. The schedules for these days are provided as Annexe A, but it is useful to pick out some activities that helped assure as reliable and valid a study as was possible given the circumstances of wide variation in CELs’ previous contact and experience in education research. Research workshop 1 (27th September 2010) The aim of the day was to provide CELs with background to the project and with particular experience at designing and using an on-line survey to collect evidence of experiences and impacts of CPD courses. A second aim was to decide on the selection of courses for collection of data. A morning session was provided for CELs to experience and practise the use of the question design tools of the Bristol On-line Survey (BOS) package. In the afternoon CELs contributed to refining the survey instrument for the pilot study, annotating a first version. In this way CELs gained some ownership of instrument design as well as expertise at using BOS for future use in their centres. The third session of the day allowed CELs to refine and share, as a team, the selection of courses for data collection using the BOS and case studies. In preparation for the next workshop, which was purposely scheduled to take place after completion of the on-line data collection, CELs were asked to use the BOS package to compile frequencies of questionnaire responses of three specific items, for each Mode and Theme. CELs were requested to report their findings at the next workshop comparing and interpreting response patterns for respondents from their SLC versus those for all respondents. 12

Research workshop 2 (15th November 2010) As shown in the attached schedule (Annexe A) the morning activity was to have been allocated to analysis of data collected using the pilot version of the BOS instrument agreed following the previous workshop. These activities were curtailed, however, by extensive discussions on the modes of courses selected for study at each centre. In the afternoon CELs engaged with activities to help collect and analyse information from interviews in case study schools. An activity to help improve reliability was used where CELs were given transcripts from interviews carried out in the previous study at York and asked to extract and allocate examples from responses according to the analysis framework they would use for real in this study (see Annexe D). When they had completed this task, CELs shared their allocations and were provided with the York team’s view for the same extracts and there was discussion as to any variances. The exercise was carried out to arrive at a set of commonly agreed strategies for extracting and ascribing items from interviews. Extensive guidance notes (also shown in Annexe D) providing definitions for sections and themes of the framework were provided to help consistency in decisions used to construct CEL research reports.

3.2.4.

The case studies

Case studies involved face-to-face interviews with teachers from two different schools attending courses agreed to be studied following stage 6 of the selection procedures described in section 3.2.1. In secondary schools the intention was to also interview a Head of Department or subject and one other senior manager, usually the person managing and monitoring CPD activity for the whole school. In primary schools the intention was to interview the Headteacher as well as the teacher attending a course. The interview schedules for teacher participants and their managers are appended as Annexe C. Interviews were recorded and transcribed and the text of each interview was allocated line numbers. At the second research workshop in November, CELs were introduced to and trained to apply an analysis schedule. CELs were required to analyse text and extract examples of responses according to an a priori set of themes agreed by the central research team and that emerged from the previous study (Bennett, Braund and Lubben, 2010). The analysis schedule is appended as Annexe D. CELs were required to complete a research report for studies at their centres that included the analysis described above, background details of the courses and their content and some details of schools, such as from OfSTED reports. The CELs’ reports were scrutinised by members of the central research team at York by a process of exhaustive analysis that added to CELs analyses until no more relevant details or significant quotes from interviews emerged. This final analysis was used to construct the findings section on case studies in this report (Section 5). It should be noted that not all centres were able to provide data for two case studies per course. Two cases were reported for technicians and one for a teaching assistant and these were rejected on the grounds that they lay outside the remit of the study and in any case provided few insights on the impact of CPD on teachers’ practices or pupils’ learning. Where there were two teachers from the same school these were treated as one case as in the previous study (and data from them labelled as being from teacher a or teacher b). 13

After scrutiny of reports returned from the CELs, the final number of case studies from which data could be drawn was fourteen rather than the twenty that had been intended. These cases are listed in Table 2 by centre, course title, mode and theme. Table 2 also shows details of the interviews carried out for each case. In all but one case (Teacher 2 in the Yorks and Humberside study) there are data available from an interview with the teacher participant and at least one other person in the school. In five of the fourteen cases data were available from interviews with three people in the school, as was the original intention. It should be noted that following these discussions and subsequent changes to the modes linked with selected courses, only Mode A (single contact CPD) and Mode C (multi-contact CPD) ‘survived’ from the original set of four and so these were the two analysed for the case studies in this report. It was possible, however, to collect data for all four modes using the online survey of participants’ views and experiences of CPD.

3.2.5.

Anonymity and codes used in the case studies

Throughout the report the identity of schools and persons has been anonymised in line with best practice in educational research. Thus schools and teachers are referred to by fictitious names or letters and interview data by codes referring to the role of the person interviewed: T= SMT or HT = HoD =

participant teacher member of school’s senior management responsible for CPD (HT is Headteacher in the case of primary schools) Head of Science

Extracts from interviews have a unique code identifying the centre, case, person interviewed and the line number of the transcribed text from which the extract came. Thus: EE/T1 [15-28] indicates that the extract is from lines 15-28 of the transcribed text of the interview with a teacher who was the first case (school) for the course studied by the East of England SLC.

14

Table 2. Cases in the evaluation [Mode A = single contact CPD; Mode C = multi-contact CPD. (P) = Primary, (S) = Secondary] Course Evaluated

Cases

Centre

Title

Theme

Mode and place

Participants

Others interviewed

North East

Outdoor Learning: Using the School Grounds (P)

Pedagogy

A (1 day, at SLC)

NE/T1 at ‘W’ Primary School

NE/SMT1 (Science Coord.)

London

Heads of Science: Leading and managing the science department (S)

Leadership

C (2 days 8 months apart with online forum, at SLC)

L/T1 at ‘On 2 sites school’ L/T2 at ‘Academy’

L/SMT1 (Line manager) L/SMT2 (Asst Principal)

East Midlands

Triple Science: chemistry + raising attainment (S)

Curriculum

A (1 day, at SLC)

EM/T1 at ‘South Vale College’

EM/HoD1 (Head of Science) EM/SMT1 (Vice Principal i.c. CPD)

East of England

Pedagogy

North West

Assessment for Learning in the Science Classroom (S) How Science Works (S)

EE/T1 at ‘Harold High’ EE/T2 at ‘Hellington High’ NW/T1a and NW/T1b both at ‘Stringbourne High School’

EE/SMT1 (Assistant HT i.c. CPD) and EM/HoD1 EE/HoD2 NW/HoD1ab (same as T1b) NW/SMT1ab (SLT member)

South East

Active approaches to A-level Chemistry (S)

Pedagogy

C (2 days a month apart, at SLC) C (3 half days, one each term, at LEA outreach centre) A (1 day, at SLC)

SE/T1 at ‘King John College’

SE/SMT1 (Head of Curriculum)

South West

Successful Science and Engineering Clubs (S) Enhancing Science with ICT (S)

Curriculum

A (1 day, at SLC)

SW/T1 at ‘Broadside School’

SW/HoD1 (Head of Science)

Pedagogy

C (2 half days four months apart, at the school)

WM/T1a and WM/T1b both at ‘Western High School’

WM/HoD1ab (Head of Science) WM/SMT1ab (Asst Head, i.c. CPD)

Yorks and Humber

Developing the role of the science subject leader (P)

Leadership

C (2 days 2 months apart, at the SLC)

YH/T1 at ‘St Martins P.S.’ YH/T2 at ‘Beaton P.S.’

YH/HT1 (Headteacher)

NSLC at York

New and Aspiring Heads of Science (S)

Leadership

C (3 x 3 days over 4 terms, residential at the NSLC)

NSLC/T1 at ‘Granton School’

NSLC/HoD1 (Head of Science) and NSLC/SMT1 (Assistant HT for CPD) NSLC/HoD2; NLSC/SMT2a (line manager) and NSLC/SMT2b (Asst HT CPD)

West Midlands

Curriculum

NSLC/T2 at ‘Samuel Sanders’

15

4.0

Findings (1): the online survey of course participants

Questionnaire responses have been analysed in order to answer the following three research questions: 1. 2. 3.

How do schools approach their planning of professional development given the choices of modes of PD available? What are the strengths and weaknesses of different modes of PD in initiating and sustaining change in practice? Which particular modes of PD are best suited to a particular type of action of change, such as the improvement of subject pedagogy, curriculum development, or leadership capacity?

The responses of 150 participants have been grouped according to the mode of the CPD course the respondent experienced. The Modes (A-D) have been defined as follows: A. B. C. D.

One day (or single contact) interactive CPD sessions at SLCs or other ‘off-site’ facilities. Residential or multi-day CPD at SLCs or other ‘off-site’ facilities. Sequences of provision (multi-contact) with an expectation of some tasks in between. Bespoke, in-school delivery of a core or unique course where participants have a clear common background.

Similarly, the 150 responses have been grouped according to the theme of the CPD course the participant took, i.e. within the theme of improving Pedagogy, the Curriculum or Leadership capacity, as previously defined. The stratified sampling of courses according to their mode of PD and themes resulted in a reasonably balanced set of responses across modes and themes. However, only after data collection was completed was it realised that mode allocation of courses had to be adjusted, as several CELs had changed their opinion about the mode of the courses delivered at their SLC. Thus, within the sample there is now little independence between mode and theme. For instance, almost all respondents taking a Mode D course aim at improving their Leadership capacity. Thus, the last research question can only be addressed indirectly. Since a very small number of respondents (8) took a Mode B course, these respondents have been presented in the results below, but their data have not been analysed for mode-characteristics.

4.1

Selection and planning of professional development

Three aspects of CPD selection and planning are presented, i.e. the expectations of participants of the courses they took; the factors that influenced their choice of CPD; and, specifically, the influence of the mode of CPD delivery on their choice.

16

4.1.1

Expectations of PD courses

Teachers were asked to indicate the expectation they had before the course in terms of six possible outcomes. Table 3 provides a summary of the responses differentiated according to the groups of teachers taking the different Mode and Themes of CPD courses. Table 3: Expected outcomes of CPD courses for the various Modes and Themes (n=150) Mode (%)

Theme (%)

A (n=52 )

B (n=8 )

C (n=60 )

D (n=30 )

Ped (n = 27)

Curr (n = 61)

Lead (n = 62)

All (%) (n=150)

Resources or activities to use in my teaching:

92% (48)

75% (6)

80% (48)

73% (22)

89% (24)

89% (54)

73% (45)

83% (124)

Knowledge of teaching strategies

75% (39)

50% (4)

82% (49)

73% (22)

74% (20)

77% (47)

76% (47)

76% (114)

Examples to help my colleague(s):

52% (27)

50% (4)

58% (35)

70% (21)

56% (15)

53% (32)

66% (41)

58% (87)

Networking with other teachers

37% (19)

25% (2)

57% (34)

57% (17)

30% (8)

53% (32)

53% (33)

48% (72)

Subject knowledge

40% (21)

50% (4)

35% (21)

47% (14)

41% (11)

36% (22)

44% (27)

40% (60)

14% (7)

25% (2)

53% (32)

33% (10)

19% (5)

18% (11)

56% (35)

34% (51)

Expectation:

Answering questions about policy or practice:

The last column of Table 3 above shows that more than three out of four teachers expected the CPD course to produce new teaching resources and familiarity with new teaching strategies (83% and 76%, respectively). Around half of the teachers expected the CPD course to provide examples of either of these for dissemination to colleagues, and for opportunities for networking (58% and 48%, respectively). Just over one in three teachers expected the CPD experience to improve their subject knowledge, or answer queries regarding education policy or practice (40% and 34%, respectively). Data from a separate questionnaire item showed that virtually all teachers (97%) felt that their expectations were met by the CPD experience, thus the expected and actual outcomes match very closely. The expectations of teachers taking Mode A courses focussed more commonly on the provision of teaching resources (92% vs 83%), and considerably less so on networking (37% vs 48%) or on obtaining answers for policy and practice issues (14% vs 34%). The expectations of teachers of Mode C courses were a complete contrast. They focussed more usually on networking (57% vs 48%), and on resolving policy/practice issues (53% vs 34%). Teachers taking Mode D courses more usually expected to gain exemplary teaching materials and strategies for dissemination to peers (70% vs 58%), develop teacher networks (57% vs 48%), but fewer than average prioritised obtaining new teaching resources (73% vs 83%). 17

Several differences in expectations emerged between teachers taking courses in the three different themes. Teachers taking a CPD course strengthening Pedagogy expected to obtain novel teaching materials (89% vs 83%), and fewer than average expected to be able to network (30% vs 48%) or a respond to policy/practice issues (19% vs 34%). Those teachers on courses with a Curriculum theme showed a similar pattern. Both matched the priorities of the group taking Mode A courses. In contrast, teachers on the Leadership courses more commonly expected to obtain example materials and strategies for helping colleagues (66% vs 58%) and suggestions on policy and practice issues (58% vs 34%), and their expectations gave a lower priority to teaching resources (73% vs 83%) as an outcome of the CPD experience. This matched the pattern for Mode D respondents.

4.1.2 Factors influencing the choice of PD course Table 4 below summarises the teachers’ responses regarding six potential factors that influenced their choice of the PD course they took. The factors are sequenced in order of perceived importance. Table 4: Self-reported factors influencing teachers’ choice of CPD (n=150) The CPD experience: Rating

Could address CPD needs identified by me

Could allow me to help teachers in my Dept or Key Stage

Fits with the school's development plan

Could allow me to help change school/dept policy

Was suggested by a senior colleague

Was recommended by a colleague or a friend

Very important

79% (118)

49% (74)

31% (47)

23% (34)

21% (32)

13% (20)

Quite Important

17% (26)

41% (61)

55% (83)

59% (88)

55% (82)

48% (72)

Not very important

2% (3)

7% (10)

9% (14)

13% (19)

16% (24)

26% (39)

Not important

2% (3)

3% (5)

4% (6)

6% (9)

8% (12)

13% (19)

The data in Table 4 show that the choice of CPD was influenced by a complex set of factors but was mainly self-directed, i.e. influenced by factors related to the teacher. For almost all teachers the choice was determined by the CPD needs they had identified themselves: 96% considered this a very important or quite important influence. The potential for supporting others through the CPD influenced the CPD choice for 90% of the respondents, whereas policy related-issues (the fit with the school development plan, and a possible contribution to a change in school policy) were considerations for 86% and 82% of the teachers, respectively. Suggestions by others, e.g. by senior colleagues and particular by peers and friends, had less influence (only 76% and 61% considered this to be a factor in their choice). Table 5 presents the influence of each of the six factors on the choice of the Modes of PD. Numbers of teachers are reported that considered the respective factors very important or quite important in their choice of CPD.

18

Table 5: Percentages of teachers per Mode considering factors an important influence on CPD choice The choice of CPD was influenced by the fact that the CPD experience:

Rating

Could address CPD needs identified by me

Could allow me to help teachers in my Dept or Key Stage

Fits with the school’s development plan

Could allow me to help change school/dept policy

Was suggested by a senior colleague

Was recommended by a colleague or a friend

Mode A (n=52)

96% (50)

94% (49)

88% (47)

78% (40)

82% (42)

63% (33)

Mode B (n=8)

100% (8)

88% (7)

88% (7)

88% (7)

63% (5)

38% (3)

Mode C (n=60)

98% (59)

90% (54)

85% (51)

82% (49)

73% (44)

60% (36)

Mode D (n=30)

90% (27)

83% (25)

83% (25)

87% (26)

77% (23)

67% (20)

All modes (n=150)

95% (144)

90% (135)

86% (130)

82% (122)

76% (114)

61% (92)

The ranking of the importance of the six factors was the same for all modes of PD. Thus, the different factors did not precipitate PD choices towards a specific mode. Only for Mode D participants did the weighing of factors contributing to the CPD decision vary partially from the average. A less than average proportion of these participants selected the CPD course on the basis of their own identified PD needs, or based on the opportunity it provided to help teachers in their department/school. A larger than average proportion of these teachers selected the CPD course because of its potential to influence school policy, and on the strength of the recommendation of a colleague. These findings tally with the nature of Mode D provision: it is school-base PD provision, tailor-made for the school staff and thus less likely to be selected by the individual, but rather by colleagues.

4.1.3

Perceived importance of the Mode of PD on CPD choice

Respondents were asked how important the mode of CPD (one day, several days, two blocks [multicontact], residential, twilight session, online training) was in the selection of their course. Table 6 below provides a summary of the responses differentiated for groups of teachers taking each mode.

19

Table 6: Importance of the Mode of PD in selecting a CPD course by Mode Mode (%) All (%) (n=150)

A (n=52 ) 54% (28)

B (n=8 ) 38% (3)

C (n=60) 52% (31)

D (n=30 ) 40% (12)

Quite Important

31% (16)

38% (3)

33% (20)

40% (12)

34% (51)

Not very important

12% (6)

25% (2)

10% (6)

13% (4)

12% (18)

Not important

4% (2)

0% (0)

5% (3)

7% (2)

5% (7)

Rating Very important

49% (74)

The data in the last column of Table 6 show that a vast majority (83%) of the respondents considered the mode of CPD quite important or very important in their selection of the CPD course. For different modes of CPD (A-D), the proportion of teachers finding the mode characteristics important only varied between 80-85%. However, the reasons for selecting the respective modes varied considerably. Table 7 presents the criteria for judging the suitability of the Mode for those teachers (n=125) who identified the Mode as an important factor in their selection of the CPD course.

Table 7: Criteria for considering the Mode of PD important in the choice of CPD (n=125) Mode (%) A (n=44 ) 75% (33)

B (n= 6) 50% (3)

C (n= 51) 53% (27)

D (n= 24) 29% (7)

All (%) (n=125)

Best use of my time

66% (29)

50% (3)

43% (22)

54% (13)

54% (67)

Provides the most thorough learning

25% (11)

50% (3)

45% (23)

46% (11)

38% (48)

Value for money

27% (12)

33% (2)

26% (13)

38% (9)

29% (36)

Fit with my personal/ family life

25% (11)

50% (3)

22% (11)

17% (4)

23% (29)

Criteria for selecting this mode Least disruption to my teaching

56% (70)

The last column of Table 7 suggests that of those teachers indicating that the CPD mode is an important determinant for course selection (n=125), more than half used selection criteria around minimising the disruption to teaching (56%), and the efficiency of a teacher’s time (54%). About one in three teachers selected the CPD mode on the basis of the expected quality of the learning

20

experience (38%). Lastly, about one in four teachers took account of the perceived value for money of the CPD mode (29%) and the way the CPD mode fitted with their personal/family life (23%). The other columns in the same Table suggest that a larger than average proportion of teachers selected a Mode A course because these courses are seen as causing minimal disruption to teaching, and providing best use of time. Mode A teachers used the criterion of an expected thorough learning experience less often than average. In contrast, teachers who selected a Mode C course more commonly on the basis of an expected thorough learning experience, are less concerned about its perceived best use of time. A more than average proportion of teachers selected a Mode D course because of its perceived value for money and the expected thoroughness of the learning experience. Teachers opting for a Mode D course used the criterion of fit with personal/family life or minimising disruption to their teaching less frequently.

4.1.4

Strengths and weaknesses of Modes of PD in initiating change in practice

The impact of the CPD experience will be reported at four levels of Guskey’s (2002) Impact Model. Firstly, the extent to which the experience challenged the respondents’ way of teaching (level 2) is reported. Secondly, the way in which the CPD experience is disseminated beyond the participant and his/her pupils (level 3). Both the impact on participants’ use of teaching materials and teaching strategies in the classroom are presented (level 4). Lastly, the presumed change in pupils’ abilities (level 5) as a result of the CPD experience is presented.

4.2

Impact of Modes of PD at Level 2

Table 8 below summarises the self-reported level of challenge the CPD course provided to the way respondents teach. Since this challenge relates to teachers’ understanding and attitude, this impact is at Guskey’s level 2. Table 8: Teachers’ reported challenge of the CPD to their way of teaching (n=150) Mode (%)

Theme (%) All (%) (n=150)

A (n=52 ) 12% (6)

B (n=8 ) 0% (0)

C (n=60 ) 22% (13)

D (n=30 ) 13% (4)

Ped (n = 27) 15% (4)

Curr (n = 61) 8% (5)

Lead (n = 62) 23% (14)

To some extent

67% (35)

100% (8)

67% (40)

70% (21)

67% (18)

69% (42)

71% (44)

69% (104)

Not much/ not at all

21% (11)

0% (0)

12% (7)

17% (5)

19% (5)

23% (14)

7% (4)

15% (23)

Challenge Very much

15% (23)

The last column in the Table shows that about two in three teachers (69%) felt their teaching was challenged at least to some extent, and that one in six teachers felt very much challenged, or not challenged at all (15% each). The Table also shows that only small differences emerged between the challenges to teaching provided by the various CPD modes, apart from a higher challenge for Mode C courses especially compared to Mode A courses. Surprisingly, courses within the Leadership theme

21

have provided a higher challenge to established teaching methods than courses aimed at improving the curriculum and even compared to those improving pedagogy.

4.2.1

Impact of Modes of PD at Level 3

Data were collected on the wider impact of the CPD experience affecting the school community and beyond thus opening a window on impact at Guskey’s level 3. The two Tables below present data on the reported activities undertaken to disseminate the gains from the CPD experience (Table 9) and the perceived secondary beneficiaries. Table 9: Reported dissemination activities after CPD (n=150)

Activity Meeting with colleagues in department or key stage

A (n=52 ) 33% (17)

Mode (%) B C (n=8 ) (n=60 ) 25% 67% (2) (40)

D (n=30 ) 47% (14)

Theme (%) Ped Curr Lead (n = 27) (n = 61) (n = 62) 48% 39% 58% (13) (24) (36)

All (%) (n=150) 49% (73)

Chat with another teacher

63% (33)

38% (3)

48% (29)

30% (9)

41% (11)

64% (39)

37% (23)

49% (74)

Whole school or department training event

15% (8)

75% (6)

35% (21)

17% (5)

11% (3)

15% (9)

45% (28)

27% (40)

Putting teacher resources onto school’s VLE/intranet

17% (9)

25% (2)

27% (16)

23% (7)

15% (4)

26% (16)

23% (14)

23% (34)

Production of new training materials

10% (5)

50% (4)

25% (15)

20% (6)

7% (2)

11% (7)

32% (20)

19% (30)

Nothing

8% (4)

(0)

3% (2)

6% (2)

7% (2)

8% (5)

2% (1)

6% (8)

Table 9 shows that only in a very few cases (6%) were no efforts made to disseminate the CPD experiences. About half of the participants (49%) disseminated their CPD experiences through reports in meetings of their departmental or key stage team. The same proportion shared these experiences through informal chats with colleagues. About one in four participants organised wholeschool or department training events (27%), or made teacher resources available to colleagues in electronic format (23%). A slightly smaller proportion (19%) of the respondents produced new training materials to disseminate the CPD messages. It is evident that any of these dissemination methods (apart from an informal chat with colleagues) was used less by participants in Mode A, Curriculum and Pedagogy courses. Mode C courses (and Leadership courses) showed a higher than average dissemination through reporting in meetings, whole school training events and the production of training materials. Mode D participants organised less whole school training events and disseminated less through informal chats to colleagues, since dissemination had supposedly already been built into the Mode of the CPD.

22

In parallel, respondents were asked who they considered to have benefitted from their CPD experience. Table 10 below shows the reported beneficiaries apart from themselves and their pupils. Table 10: Beneficiaries of the CPD apart from the participants and their own pupils (n=150)

Challenge The whole department or Key Stage team One or two other teachers Teachers in other schools The whole school Student teachers Teachers in other departments Nobody else

A (n=52 ) 33% (17)

Mode (%) B C (n=8 ) (n=60 ) 50% 68% (4) (41)

D (n=30 ) 30% (9)

Ped (n = 27) 56% (15)

Theme (%) Curr (n = 61) 36% (22)

Lead (n = 62) 55% (34)

All (%) (n=150) 47% (71)

52% (27)

(0)

20% (12)

30% (9)

41% (11)

52% (32)

8% (5)

32% (48)

17% (9)

(0)

13% (7)

37% (11)

7% (2)

21% (13)

18% (11)

18% (27)

6% (3) 10% (5) 6% (3)

63% (5) (0) (0)

17% (10) 15% (9) 10% (6)

13% (4) 17% (5) 7% (2)

7% (2) 7% (2) 4% (1)

2% (1) 11% (7) 11% (7)

31% (19) 15% (9) 5% (3)

15% (22) 12% (19) 7% (11)

6% (3)

(0)

2% (1)

3% (1)

(0)

7% (4)

2% (1)

3% (5)

The data in Table 10 show that hardly any (3%) of the respondents saw themselves and their pupils as the only beneficiaries of the CPD experience. Almost half (47%) of the respondents reported that their whole department or Key Stage team benefitted from their CPD, whereas one in three (32%) teachers reckoned that only one or two colleagues benefitted. In addition, one in six teachers reckoned that the PD had a positive impact on their whole school, teachers in other schools and student teachers. This ranking mirrors the dissemination activities reported in Table 9. The reported benefits for whole departments or Key Stage teams and for whole schools are particularly prevalent for Mode C and Leadership courses (as expected). This is the opposite for Mode A courses, and for courses directed at improving the Curriculum and Pedagogy. Instead, these courses reportedly tended to benefit only one or two other teachers. It is striking that Mode D courses in particular benefited teachers in other schools, presumably within a cluster context.

4.2.2

Impact of Modes of PD at Level 4

The two tables below summarise the teachers’ reported classroom change due to the CPD experiences, thus reflecting the impact at Guskey’s level 4. The first Table (11) provides information about the classroom usage of teaching materials promoted during the CPD course. The second Table (12) provides a summary of reported changes in teaching strategies in the classroom.

23

Table 11: Teachers’ reported immediacy of usage of CPD generated teaching materials (n=150) Mode (%) Time of use of materials

Theme (%) All (%) (n=150)

Used straight away or in a few weeks

A (n=52 ) 58% (30)

B (n=8 ) 88% (7)

C (n=60 ) 87% (52)

D (n=30 ) 67% (20)

Ped (n = 27) 70% (19)

Curr (n = 61) 61% (37)

Lead (n = 62) 85% (53)

Will be used later in the school year

23% (12)

13% (1)

12% (8)

23% (7)

19% (5)

24% (15)

11% (7)

19% (28)

Will rarely or never be used

19% (10)

0% (0)

(0)

10% (3)

11% (3)

15% (9)

3% (2)

9% (13)

73% (109)

Table 11 suggests that almost three in four teachers (73%) used teaching materials promoted in the CPD course within a few weeks after completion of the course. This proportion was considerably higher for teachers on Mode C courses and Leadership courses. It also shows that an additional 19% of teachers expected to be using the materials within a year after the course. Less than one in ten teachers (9%) reported that materials will rarely or never be used, although the proportion of nonusers was higher for Mode A courses, and those directed to strengthening the Curriculum. Table 12: Teachers’ reported changes in their classroom teaching (n=150) Mode (%) B C (n=8 ) (n=60 )

D (n=30 )

Ped (n = 27)

Theme (%) Curr (n = 61)

12% (7)

17% (5)

11% (3)

7% (4)

16% (10)

11% (17)

88% (7)

72% (43)

37% (11)

56% (15)

36% (22)

69% (43)

53% (80)

37% (19)

0% (0)

7% (4)

30% (9)

22% (6)

39% (24)

3% (2)

21% (32)

19% (10)

0% (0)

10% (6)

17% (5)

11% (3)

18% (11)

11% (7)

14% (21)

Change in classroom teaching Most of my lessons

A (n=52 ) 8% (4)

13% (1)

Some of my lessons

37% (19)

A few of my lessons Not really changed the way I teach

Lead (n = 62)

All (%) (n=150)

Table 12 indicates that two out of three teachers reported that the CPD had affected their teaching in some lessons or even most lessons (53% and 11%, respectively, thus totalling 64%). The proportion of teachers reporting such change in teaching was considerably higher for Mode C courses (a total of 84%) and, surprisingly, for Leadership courses (85%). However, such change in teaching was less than average for teachers taking Mode A courses (totalling 45%) and Curriculum courses (totalling 43%). It was noticeable that courses specifically directed at improving pedagogy only reported an average change in classroom teaching, i.e. for 67% of these teachers.

4.2.3

Impact of Modes of PD at Level 5

Respondents were asked if the CPD had an impact on four pupil abilities, i.e. their science skills (e.g. measurement, graphing, investigation, and planning skills); their general skills (e.g. thinking, communication or problem solving skills); their understanding of science concepts; and their 24

attitudes to science. Since these data report pupil change, the responses reflect impact at Guskey’s level 5. It is notable that for the impact of each of these pupil abilities half or more than half (50-57%) of the respondents indicated that they were uncertain of impact. Table 13 below presents the number of participants reporting positive improvement in each of these pupil abilities per PD Mode and Theme, and Table 14 summarises the evidence respondents depended on for their judgements, again per mode and theme. Table 13: Impact of CPD Mode and Theme on pupils’ abilities Mode (%) Improvement in:

Theme (%)

All (%) (n=150)

A (n=52 ) 21% (11)

B (n=8 ) 25% (2)

C (n=60 ) 38% (23)

D (n=30 ) 27% (8)

Ped (n = 27) 26% (7)

Curr (n = 61) 23% (14)

Lead (n = 62) 36% (22)

General skills

33% (17)

13% (1)

48% (29)

23% (7)

33% (9)

33% (20)

44% (26)

36% (54)

Understanding of science concepts

40% (21)

50% (4)

40% (24)

30% (9)

44% (12)

37% (23)

37% (23)

39% (58)

Attitudes to science

33% (17)

38% (3)

40% (24)

30% (9)

41% (11)

32% (20)

36% (22)

35% (53)

Science skills

29% (44)

The data indicate that for each of the four pupil abilities roughly one out of three participants reported an improvement due to the CPD. Improvements in science skills were more common after Mode C and Leadership course, and distinctly less so for Mode A courses. Equally, improvements in pupils’ general skills were reported more often for Mode C and Leadership courses, and less so for Mode D courses. An improvement in attitudes to science was relatively more frequent for Mode C and Pedagogy courses. These interpretations should be made in the light of the fact that about half of the respondents indicated that they were not sure about impact. The summary in Table 14 sheds some light on the strength of the evidence teachers used when reporting change in any of their pupils’ abilities. For each mode and theme the Table presents the total number of instances where evidence was across the four abilities, and the number of these that are based on the various sources of evidence: tests; homework or class tasks; pupil questionnaires; observation of lessons by the participant, or by another person. Where teachers cited evidence for change, over half of them referred to their own observations in their lessons (56%) and to the quality of pupils’ homework or classroom tasks (52%). Only one in four respondents referred to more objective evidence, such as lesson observations by a colleague (28%), pupil questionnaires (26%), or test results (20%). The use of tests was more common for Mode C, and the use of pupil questionnaires and outsider lesson observations were more usual for Mode D courses. It was notable that changes due to the courses intending to improve Pedagogy and Curriculum were, generally, less well evidenced.

25

Table 14: evidence of change in pupils’ skills, understanding of science concepts and attitudes Mode (%) Evidence of change from:

Theme (%)

A (n=66 )

B (n=10 )

C (n=100 )

D (n=33 )

Ped (n = 39)

Curr (n = 77)

Lead (n = 93)

All (%) (n=209)

Tests

12% (8)

(0)

27% (27)

21% (7)

15% (6)

14% (11)

25% (23)

20% (42)

Homework/class tasks

56% (37)

40% (4)

53% (53)

45% (15)

49% (19)

55% (42)

52% (48)

52% (109)

Pupil questionnaire

11% (7)

30% (3)

29% (29)

45% (15)

18% (7)

10% (8)

41% (38)

26% (54)

Lessons observed by me

59% (39)

60% (6)

56% (56)

70% (23)

54% (21)

45% (35)

66% (61)

56% (118)

Lessons observed by another person

14% (9)

30% (3)

32% (32)

42% (14)

33% (13)

16% (12)

35% (33)

28% (58)

4.3

Suitability of Modes for specific Themes

The sample of responses does not allow an analysis of the most suitable modes of PD for specific themes of courses since the two dimensions (Mode and Theme) are not independent. However, some indication of suitability may be deduced by returning to responses to some items indicative for the respective themes. For instance, the impact of the various modes of PD on a change in classroom teaching (as reflected in Table 12 above) may be equated to measure impact on pedagogy. Following this assumption, the data show that Mode C leads to the largest change in teaching strategies, even though most of the Mode C participants participated in courses intended to improve Leadership. Similarly, the immediacy and adoption rate of new teaching materials (as reported in Table 11) may be taken as an indication of impact in the area of Curriculum change. Here again, the uptake of teaching materials is most prevalent amongst Mode C participants. Lastly, the impact in the area of Leadership may be indicated by the dissemination activities undertaken as a follow-up of the CPD experience (as in Table 9) and the type and range of beneficiaries of the CPD experience (as presented in Table 10). Most dissemination through contributions to departmental meetings, organisation of departmental training events and the development of training materials emerged from Mode C participants. This prominence of Mode C courses tallies with the reported larger benefit for the school or department accruing from participants in Mode C courses. However, if leadership refers to out-of-school leadership such as in school clusters, mode D courses may be most appropriate.

26

5.0

Findings (2): the Case Studies

Findings from analyses of case studies are reported first in terms of the research questions of the study: 1. 2.

3.

What are the strengths and weaknesses of different modes of professional development in initiating and sustaining change in practice? Which particular modes of professional development are best suited to a particular type of action or change, such as development of leadership, curriculum development, or subject pedagogy? How do schools approach their planning and professional development given the choices available?

Later, case study outcomes are considered in relation to the ‘routes model’ of professional development impact and the concept of schools’ ‘implementation space’ for CPD.

5.1.1 What are the strengths and weaknesses of different modes of professional development in initiating and sustaining change in practice? In case studies it was possible for two modes of professional development to be studied; single contact (Mode A: often one day) and multi-contact (Mode C: two or more CPD occasions).

5.1.1. Single contact CPD (Mode A) All four courses evaluated were of single day duration and took place at SLCs. Comments were also collected from participants in multi-event cases, as questions about strengths and weaknesses of modes of course delivery were asked of all participants, irrespective of the mode of courses they attended. One day courses were most commonly valued for minimising disruption to teaching. When centrebased, courses were seen to provide a space for concentration away from the distractions of school duties. “I prefer one day courses because I don’t like to be away from my classroom for too long . . . I feel that the kids need me . . . I need to be there to help them progress. So I prefer a one day course where there’s going to be a minimal impact on my classes.” EM/T1 [49-52] “I know there have been some examples of courses that are very good, but have involved a sort of continual period of time out of lessons over a length of time. It’s very difficult to justify that because it would have a large impact on the teaching and learning within those classes and a knock on impact on the whole faculty. You know, I think it’s difficult to convince staff to go on those sorts of courses.” SW/HoD1 [191-195]

27

The following segment from an interview with two senior managers in a 6th form college provides additional evidence of these advantages: HC:

“You certainly need a day out to not get involved in … college life. You need to go away and … I think, you know, it’s good to be able to reflect on what you do and then come back. I think half-day events are perhaps less effective sometimes, than whole day events.

SMT: Yes, sometimes we host them [CPD events] here and that’s a double-edged sword … because obviously people then, as soon as they are here, people do get dragged away. HC:

Mm.

SMT: So, just the very physical act of people going away from college can be quite valuable for them, really, so… INT:

Do you feel that one-day courses are maybe better because of time constraints or would you be happy for people to go on residential . . .?

SMT: We would be. They don’t come up very often and I think people are very reluctant to say that they want to do them because they are just worried, you know… all the students that we teach are examined, you know. In a school you’ve got your lower years which you can maybe leave but, for us, time is so tight that people are very reluctant to leave their classes for more than one lesson, which is what it would be.” SE/SMT1ab [166-185]

Indeed there may be a negative educational cost benefit in terms of the effort and time needed to organise for absence to attend half-day or twilight CPD compared with whole day events: “I think the whole problem with anything like this is you have to put in a certain amount of time to organise going out of school and so, if it’s a whole day, then it’s worth it . . . You get a benefit, but if it’s a half day, it’s less worthwhile because you still have to put in the same effort to get the time off school and you’ve still got pretty much the same amount of disruption because, depending on when your lessons start, the whole day’s gone. Twilight’s I avoid because, again you’re disrupting more days for the same amount of benefit . . . Things that are for the whole week sound fantastic but in reality the chances of being able to do it is slim.” L/T1 [175-184]

Even when some school managers prefer multi-contact CPD, the educational benefit of single day courses may still be valued. In the comment below, one Head of Department seemed to think one day courses are acceptable only for outcomes perceived to have small impact.

“You know . . . still having said that about not going out . . . I would still always support staff to take CPD where the impact isn’t going to be huge, because the benefit of that (one day course) can outweigh the cost more often than not.”

28

SW/HoD1 [181-3]

This notion of the ‘size’ (importance or scale) of the initiative relating to the duration of the course was expressed in another case: “Yes . . . I think for some major, major developments, you would consider maybe sort of allocating more than a day’s training. It’s . . . development and giving them one day’s training in CPD and expecting them to do something with that is within our expectations.” NE/SMT [60-3]

In one case, of a 6th form college, a closely located, one day course enabled all three teachers of Alevel chemistry to attend at a suitable time with little disruption to the college. Attending together seemed to facilitate follow-up and sharing of outcomes. “I was looking for some CPD for the department, so all of us went . . . the whole department went. The Impact Award was attractive in that it was no cost to the college and it was at a time of year when we have no classes, so our contact time is less.” SE/T1 [45-8; 70-73]

5.1.2 Multi-contact CPD Six courses involving more than one contact of CPD were evaluated. The majority (four) were two events, usually days, separated by varying lengths of time (see Table 1, page 2). One course involved three half days, one in each term and the remaining course involved four periods of three residential days over an 18 month, four term, period. The most commonly expressed advantages of multi-contact courses over other modes were that they provide cognitive space for deeper reflection on teaching and to test innovation and actions that help take professional learning further. “I certainly think that was one area that allowed me to develop, to continue developing a vision for the department . . . clarifying a few things in terms of curriculum. What I’m trying to say is within those two (separated) days I had a lot of time to think clearly about a longer term perspective: how we’re going to start fixing . . . some of the problems. Now, I went in with a few ideas, and the course allowed me to think about those (problems) in a bit more depth and a bit more clearly, out of the context of being in school, and so I certainly felt I was able to do that.” L/T2 [180-186] “It was better that it was over 2 days with a big gap in the middle because it gave me a chance to put into practice some of the things that had come out of the meeting and consolidate it. If I was coming back to the course I’d know what questions I wanted to ask. Y&H/T1 [37-42]

29

“It was quite nice that it was two days in that you could then go back and ask questions on the second day because, even though you have the sort of email contact later on you sort of don’t think about emailing for help, whereas when you are sitting there and everyone is sort of talking about stuff . . . you go, ‘Oh yes I remember I had this problem’.” EE/T1 [33-36]

The timing of blocks of training can affect impact. For example, in a case of leadership training for a new Head of Department, the start time of the course (in October) and reflection over a full academic year helped the participant settle into a new role for a short time before having to reflect on his own practice and involve himself in the process of sharing outcomes within the department: “It’s the timing of it. It was . . . in October. That was a good time because that gave him (the Head of Science) a settling in period of a month . . . a bit of time to reflect on his current practice and something that he could share after the course and probably build on.” L/SMT2 [88-91]

There were a few responses that suggested having tasks set between CPD sessions forced deeper reflection, enabling discussion of issues, innovation or approaches in school: “I did like the fact that it was over two sessions. I do think that was a strong point for me because I could see that I would be far more likely to do things that I was going to be forced to . . . after the first session to not just forget about it . . . that there would need to be some follow up.” L/T1 [139-141] “The fact that it was continuing learning was great. One-day courses are absolutely useless. What I found was that assignments in-between really got me deeper into it than I would have done had they not been there. I really got into it – I read the texts, which I would not have done necessarily if I hadn’t had those assignments . . .” NSLC/T1 [63-7]

There were two cases where schools particularly valued impact of multi-contact CPD over other modes but on condition these could be based in school as outreach, bespoke events rather than at a training centre or SLC. “I prefer more bespoke CPD in school with support, where it’s available. I think if you can actually have people who can come in and, you know, address your specific needs in school . . . it is superb. Particularly if it’s over a number of weeks over an hour, an hour and a half or so over a series of weeks. I do feel that you tend to get more from that than you do with a one-off day . . . no matter how good it is. Days out on CPD can generate ideas and enthusiasm . . . but if you’re not careful that can somehow get lost slightly when you get back into the day to day of school.” SW/HoD1 [47-53]

30

“For, for me, that bespoke training, where you get a department in a school together with a trainer and they fire ideas off each other is far more powerful than one person going out and then trying to sell it, because they can never replicate what they’ve seen.” WM/SMT2 [173-6]

Weaknesses of multi-contact CPD reiterated strengths of the single contact mode set out before; disruption to teaching and examination classes and financial cost. These were more keenly felt in smaller departments and schools particularly where there might be calls for two or more teachers to attend a course. “. . . it would be very difficult for us as a reasonably small faculty to justify releasing two people to go on the same CPD on the same day and almost impossible to justify doing it over a series of days because. . . we’ve got four full time science teachers here . . . I would be reluctant to do that. I would have to be persuaded that it was something really quite special” SW/HoD1 [259-263]

Twilight (after school hours) CPD events were seen as counterproductive as they require significant concentration and reflection when teachers are most tired after a day’s teaching. “To be honest … twilights create the difficulty of tiredness and being jaded and it not being the only thing on your brain, on your brain that day, you know? I think you need time to think, time to absorb, time to process, time to reflect.” NW/SMT1 [816-821]

Half-day courses are difficult for some teachers even when organised as multi-contact CPD. A teacher commented on some of the problems: “I think that sometimes when you’ve got one of those half days . . . in terms of the travelling from around here, its rushed and you’re rushing from one place to another and I don’t think you have time maybe to settle down and get into the swing of things as quickly. You come back and you’re worried about where your classes are. On a full day you’re more relaxed and maybe you’re more open to . . . the things (from the CPD) . . . but you’re not. “ NW/T2 [425-431]

5.1.3. Evidence of impact The quantity and quality of evidence of impact of CPD, for example to show that teachers had changed their practice, adopted new teaching or influenced their colleagues was sparse. Evidence of impact of CPD on pupils’ attitudes or attainment was also weak. In many cases responses to questions about impact were prefaced by aspirational phrases such as: ‘I think she might have done x or y’ or ‘I had hoped she would have talked to a or b’. It seems that, in many schools, systems to systematically monitor and record outcomes and impact of CPD are poorly developed and so comments about proven outcomes are less common than those reflecting hopes and intentions.

31

In many cases (nine of the fourteen) teachers or their managers found it hard to attribute changes in practice or advances in pupils’ attitudes or performance to professional knowledge gains. “Now I don’t doubt that some of those ideas may have come from this CPD but I couldn’t say to you now ‘that’s definitely come from here, that’s definitely come from there.’ You know, I’m sure that probably some things have come from here, but I couldn’t tell you what they are I’m afraid.” SW/HoD1 [112-115]

The following discussion between the interviewer and two members of the SMT in one of the secondary school cases illustrates some of the difficulties in measuring and distinguishing CPD impact from changes in general teaching confidence or ability, even when aware of the need to do this. HC:

“They (the teachers) are always getting better. And I think there are a range of courses and, you know, a range of things, input into that. And this would be an example of one of those courses that would impact onto just looking at how they can improve things all the time . . . how can the students learn better and what can we do? . . . how can we bring them up to date and how can we do different experiments?

INT:

Yes, so, there is not clear evidence from this particular course, but overall . . .

HC:

No, I think . . . I think evidencing the impact of any CPD activity is hard, yes.

SMT: You know it’s an ongoing hot potato, that one and obviously, as money will become tighter, it will be important for you to justify the CPD budget. But it is, as ‘H’ says, there’s a basket of things that can affect teachers and pupils. HC:

Mm.

SMT: And, not least, people feeling invested in and having that opportunity to take stock … you can’t always measure the outcome of that. But someone who’s had a chance to recharge their batteries can be as valuable as anything else . . . HC:

Mm.

SMT: So, I think it will be hard to measure, you know, to say, when we get the results, you know, well, our success rate went up by 0.5% . . . I don’t think you’d ever identify . . . that was because of this course or that input, or whatever. SE/SMT1ab [293-320]

When a teacher takes on a new role, for example as a Head of Science, attributing changes to previous CPD can be problematic. In the case of a course for new heads of department over one academic year, it was hard to distinguish specific impact of the CPD from impact that had accrued from the teacher’s personal development through having to manage other teachers. “I can’t . . . if I’m honest, say whether I genuinely think the course has made any difference to him whatsoever because it’s so difficult to extrapolate that from what was happening to him over that year anyway. What I can say is that I think he definitely

32

grew in confidence about dealing with day to day things, and I think he always had the skills to do it, but he dealt with things very well, in a very, in a clear and matter of fact way throughout the year, and still does, but I think it gave him a little bit of an edge I suspect, but it’s difficult to know whether that would have happened anyway . . .” L/SMT1 [706-12]

Gauging specific impact of science CPD can become less relevant or important to identify when teachers change roles, for example taking on pastoral or management responsibilities. “I know that she is quite keen to sort of move on and she is looking for sort of responsibility points eventually . . . She is starting to look quite hard at more of a pastoral route, which makes some of this stuff (in science) a little bit less immediately relevant.” EE/HoS1 [92-7]

On the other hand, the view of collecting CPD evidence as an imprecise activity can be changed when a teacher takes more of a management role, in this case a teacher training up a colleague to take her place as science coordinator of a primary school. “. . . A lot of the time I find it hard with the impacts – they can be a bit wishy-washy. Because I knew I was handing stuff over to (the new teacher) I thought I know how to do science but if I’m going to teach someone else to be a science coordinator I need to know how to do the coordinator bit properly myself so that part of my CPD was quite specific.” Y&H/T2 [173-7]

There was evidence of impact in courses for both modes and in all three themes but it was strongest for multi-contact CPD in the Leadership theme. As stated in the last section, reporting findings of the BOS, this is a consequence of these courses being connected – mode is not independent of theme. Leadership courses stress specific responsibilities to bring about change on a larger scale and for other teachers these courses often set specific tasks requiring evidence of impact. As already stated, it was rare for schools in this study to provide evidence of measurable changes. Against this trend, in one of the cases associated with a multi-contact Leadership course, there was an abundance of evidence of measurable changes; in pupils’ behaviour, their science results, uptake of science subjects, pupils’ ability to work independently and teachers’ use of questioning techniques. INT:

So if we talk about the actual impact rather than your expectations about it. What are you doing now in your teaching differently than you did before as a result?

T2:

The way that they delivered the course to us, I took away a lot of the activities they used to deliver it to us, as well as the content, if you know what I mean. So like, ‘Andrew ... you and your group over there and . . . this person go with that group’. You know, that moving about of people during activities is so different. What is it called? (jigsaw, snowballing expert-envoy). What else was there? There was the investigative side of science, which was introduced to us,

33

particularly into Key Stage 3 . . . Getting the kids working more independently and getting to ask questions, rather than being, you know, ‘here is your equipment, this is what you are going to do with it and that’s what you should find out. INT:

Yes. Next question is why did you have these particular things rather than something else? Or is it basically, with regard to your course, it is more or less everything that was relevant to teaching, you have used?

T2:

Everything.

INT:

Do you think so?

T2:

Everything. I mean there must have been nearly 40 sessions over the 12 days. And not one of them was irrelevant or rubbish. They were all useful to some extent.

INT:

So what impact have these changes had on pupils?

T2:

Well that is a tricky one to measure, but with us, I mean, part of the course was introducing new ideas to your Department and getting them on board.

INT:

But have you seen anything in your classes? . . . In terms of pupil’s behaviour or learning?

T2:

Well, Departmentally the behaviour is getting better and we measure that through the number of behaviour checks that you know we can say you are on check one for shouting out, talking etc things like that and obviously they go up the ladder the more things they then choose to do. We have kids who are on call from lessons you know who go out and all those stats come to me and we are seeing a steady reduction in those through the quality of teachers we have had, the lessons they are delivering and the courses being a bit more matched and kids getting in to it. So that is on the down, results are on the up.” NSLC/T2 [111-185]

“The number of kids choosing science options is increasing. indicators . . .”

They would be our

NSLC/T2 [190-191]

“. . . the Ofsted Inspector praised the fact that the kids were working so independently and the assessment was good and they said that it was the best questioning they had seen within the school.” NSLC/T2 [346-349]

In half of all cases, however, evidence of impact was circumstantial or assumed. For example in a multi-contact course on How Science Works, one of the trained teachers claimed you could tell 34

whether or not a member of the department had been on the course just by the way in which they taught, but evidence on which this was based remained very vague. “If you took a sample across the five year groups you could . . . probably plot when the teacher has been on the How Science Works workshop and then which kids were getting it and then obviously with pressures they might dip off again but then there would be another boost and so be quite cyclic. Yeah, you know it’s sort of like which teacher went and which classes are getting hit with it and stuff really.” NW/ T1 [419-424]

At the other extreme, where CPD seemed to have failed, there may still be peripheral outcomes attributable to the CPD, though not connected with the core intentions. In the case quoted below, the intention of the single contact CPD was to help schools set up Science and Engineering Clubs. The initiative in this school failed due to logistical problems in providing enough time at lunchtimes or after school for club events, but there were other gains, principally affecting liaison with primary schools and use of ‘science ambassadors’: “I’m going to use this (materials and ideas from the course on STEM clubs) with the primary school thing that I’ve talked to you about because what I’m going to do is set up a physics interactive day using our lab and different physics experiments and get our Year 9 Learning Ambassadors in here to explain it to the younger children, you know, there’s a marshmallow with using a vacuum pump to suck the air out of it … how does that happen, why does it happen, you know this kind of thing?” SW/T1 [236-243] “I mean it was through thinking about setting up the STEM club particularly that we got in contact with *STEM Ambassador’s coordinator+ and then got the ambassadors in so although we might not actually be running this as a discreet club, I think some of the avenues… you are thinking about this and going down this track, some of the avenues this has opened up for us has helped us bring some people, helped us do some of these other things. So I would say on the face of it you know, are we running the STEM club, well no, we haven’t got any sort of STEM club but, you know… you know, we are doing a lot of those things so probably actually this training did help in some regards towards that.” SW/HoD1 [142-150]

5.2

Which particular modes of professional development are best suited to a particular type of action or change, such as development of leadership, curriculum development or subject pedagogy?

Roughly equal numbers of courses were evaluated for the three CPD themes; Leadership, Pedagogy and Curriculum. There were more cases returned for Leadership courses (6 cases from 3 courses) than the other themes (5 cases from 4 courses for Pedagogy and 3 cases from 3 courses for Curriculum). As previously stated, mode cannot be separated from theme in the case of Leadership. The other two themes had examples of single and multi-contact courses but, on the small amount of evidence available, there seemed to be no difference in impact of mode within a theme. 35

Given the general problems that schools had in providing concrete, measurable examples of change attributable to CPD discussed above, it was again Leadership courses that provided more examples of sustained change. Indeed senior and middle managers often supported Leadership courses because they expected participants to contribute to or produce significant advances at department or whole school level. “What I was looking for was more leadership, more awareness of where this department needed to be going in terms of the national agenda. In terms of what is happening outside of schools, in terms of the community, what is it that we need to be doing in order to make our subjects successful, make it relevant, make sure that we are training students with the right courses to lead them on to whatever they are doing post-16 when they leave here. So in terms of us being a modern effective department facing the challenges of where we are now … that is what I wanted (her) to be able to do.” NSLC/HoD1 [131-138] “The courses that I am looking for are the ones that address the school’s needs first and foremost, that’s what I will be looking for and secondly … then it would be for the actual person themselves. Now if the two combine, then that’s great, but I’m wanting value for money basically and sometimes it might just be to develop a person in an aspect that they are weaker on but mainly I’m looking for something that can then feed back into the school to help us as a school keep moving forward and improving.” Y&H/HT1 [44-50]

In one case the Deputy Headteacher of a secondary school saw the importance of investing CPD effort in middle managers to support the work done by senior management in bringing about whole school change.

“And again we are always keen to develop our middle leaders … to make sure that they have got the skills to support the senior leadership team in making our decisions about the way the school goes. In essence, without quality middle leadership, the work that you do at senior leadership can just fall by the wayside . . .” NSLC/SMT2a [70-75]

Later, in the same interview, this Deputy provided insights into why he regarded residential multicontact CPD as a firm platform for developing the professional reflection and macro-management skills needed to guide and sustain change and lead a high performing department. SMT: A residential course kind of creates that space because often you can do a day course and then you kind of return home or you return back to school and you are straight back in to the day-to-day and you don’t have that time period (to reflect). I think with a residential it builds in that necessary reflection on your

36

own practice and also allows you to set targets for what you are going to change for when you come back. INT:

That is clear. So what were the reasons for asking ‘A’ to go on this course? What personal qualities actually were involved? When you think about ‘A’, was there anything about him that is relevant?

SMT: I think it was the need for him to be able to be more strategic in his role and I think often when people first step into a leadership role they end up micromanaging and I think you have to give them the skills, the ability to step back and I think almost give them permission to step back and be more proactive in their planning . . . I think creation of a vision, how they want the team to perform, turning that vision into sort of actions and targets that are achievable. I think that is the most important part and being able to lead a team to be able to perform.” NSLC/SMT1a [89-105 seq]

The comments above came from a case of a course with four contacts of three residential days, but the benefits of time spent away from school in leading change were also seen in courses of lesser duration. The comments below were made by the Headteacher of a primary school on seeing the effects on science development in the school of a teacher having attended a course (for primary science subject leaders) of just two single days: “I don’t see us as having barriers to be honest. I’ve got a very positive staff . . . she’s very well organised. The course has given her a very good outline of the process of the next step she needs to take. Having those days out – and its quality time out, which is the key thing, not a tag on at the end of the working day . . . she’s been able to do that high quality thinking and discussion element. She’s come back into school very organised about how we’re going to feed it back in, keep people on board, what will they need so she’s done her audits and that’s what made it successful. Its’ not as if she’s gone on a twilight and she’s come back in and said this is what we’ve got to do. So it’s the high quality approach and those aspects of the course that make it successful back in school.” Y&H/H1 [169-179]

A noticeable outcome of courses connected with pedagogy was that they provided ‘validated authority’ or confirmation of existing good practice, a feeling of ‘doing things right’. For example, in a course involving developing effective approaches using ICT, professional knowledge of the gaps there might be in teaching allowed the department to make a successful bid for additional equipment (data loggers). “And it’s partly a resource issue, which we managed to solve . . . through using ICT. This particular course, we used . . . as part of a bid to say,’ look, we’ve had some training, we want to use this (data logging) now’. And we’ve actually got some data loggers in the department, which we wouldn’t have got before.” WM/HoD1 [197-200]

37

In the case of a course on innovative approaches to teaching ‘A’-level chemistry, the Head of Chemistry, although still open to new examples, was not disappointed at finding a lot of new ideas on the course, since this endorsed existing practice. “. . . There were lots of things on the day that were good, and we already do them. So therefore they didn’t add to our palette of techniques that we could use.” SE/T1 [106-7]

“It is always disappointing not to come away with lots of new things but we’re already doing lots of the new things, so it’s not disappointing to have what we’re doing endorsed as being good practice.” SE/T1 [118-120]

The notion of ‘validated authority’ was noticed in our previous study and particularly common in pedagogy courses where teachers felt they have to try out approaches in their own classrooms before having the ‘authority’ to disseminate practices to and convince colleagues. As in the previous study this was common amongst primary teachers and occurred in this study in the case of a course on teaching science outside the classroom. “I would say myself and my class that would be what I got from the course. I would be the first port of call . . . to benefit from it and then . . . Because I’m relatively new to this, I think it’s me liking to get ideas, test the ideas to see how they work and talk to colleagues about them after I have done that.” NE/T1 [84-6]

In the case of a course in the Curriculum theme, on How Science Works, the school devised a strategy to allow all teachers in the department to attend the course provided by the SLC at a training centre used by the LEA. This involved two teachers, the Head of Department and an AST, attending the first of three half day sessions to judge quality and then asking for volunteers to attend the following two sessions. The Head of Department believed this provided a way in which the messages of curriculum change could be absorbed quickly without the need for constantly providing feedback from courses to others in the department. “Within the department we felt that it was better that everybody got a little bit rather than one person got everything.” NW/HoD1 [43-4]

“. . . but my overall vision . . . was that it would impact on the teaching and the learning . . . in the whole of the department and not just in myself. That was why I wanted everybody to get involved rather than just have one person who would come and then feedback at a departmental meeting and have little effect on anybody else.” NW/HoD1 [63-9]

38

The distinction between themes of Curriculum and Pedagogy is partly artificial. Outcomes in one theme can accrue from outcomes in the other. For example, in the course on How Science Works referred to above, a course in the Curriculum theme, the models of teaching associated with this aspect of the curriculum provided generalisable models of teaching that impacted on wider pedagogy: “Of course . . . the How Science Works evolution (of the teaching) from the course has meant that this change in methodology has . . . provided a model, because you can’t teach in the old ways . . .” NW/SMT [327-331] “Yeah . . . (there has been) a change in the methodologies in terms of teaching and learning. I have a greater overview . . . if you just teach within science you can see maybe … some impact in science . . . but it’s actually a change in the approach to teaching and learning across the school. So what’s happening is a proportion of . . . good lessons is increasing, despite the fact the OFSTED criteria changed . . .” NW/SMT [399-404]

5.3

How do schools approach their planning and professional development given the choices available?

There appear to be three main factors guiding schools’ choices of CPD for science: cost, organisation (including potential disruption) and location. The sub-sections discussing these factors are ordered according to the frequency with which factors were cited in cases as reasons influencing choice.

5.3.1. Cost Impact awards were mentioned in half of cases. Often availability of impact awards was the deciding factor over most others and even stronger than the mode (organisation of the delivery) of CPD. INT:

Were there any other deciding factors in that particular instance (of choosing the course), like location or cost or anything like that?

T1:

Well yes, the fact that we got the impact award. EE/T1 [17-30 seq]

“To be truthful . . . I think . . . the reason of choosing it was more about cost than anything … we just would not have done it without the impact award.” EE/HOD2 [42-6]

In one case the Head of Department found the impact award provided important additional time and resources to follow up and enhance impact of the CPD. “I’ve been given a certain amount of money but . . . to be honest we’ve used the impact award to allow us to have extra time after work and make the resources (from the course), which is always one of our bug bears in the past. WM/HOD [221-228]

39

Occasionally schools act as hubs or outreach centres providing SLC courses. In the case of a secondary course on AfL, this enabled a school to obtain free places for training which might not otherwise have been funded. “That is one of the reasons why we have been so keen to hold courses here for the SLC so that we can then get free places . . . I mean it doesn’t come out of my budget directly . . . they are really tight on cover and really don’t want us going out of school.” EE/HoD2 [161-167]

In another case in the East Midlands, the SMT member in charge of CPD rejected courses in London on the basis that travel would add to the total cost. This filtering would have resulted in staff never seeing what these courses had to offer. “It (literature on courses) comes to me first. I then do a sift and there is some that goes in the bin and currently, for instance, if they are in London they go in the bin, generally speaking because of cost. By the time you’ve covered a teacher for a day, you’ve paid for the course and you’ve paid £111.00 to get down to London and back . . . you don’t do it . . . you are talking about . . . £600 at least. So I do a sifting process . . .” EM/SMT [107-114]

Finally, one senior manager mentioned the increasing importance of SLC courses in a climate of diminishing pots of money that have previously been available to support CPD. “It’s going to be even harder next year. I think we already know that all the grants that I’ve managed to use and manipulate and, sort of, play with, they’re going… things like the Strategies money . . . One to One money . . . it’s all going to be put in one pot . . . it’s going to . . . devolve into the school. We know that essentially means a 10-11% cut in what we’re going to be getting. Some things will have to go . . . I had my training budget reduced in the last three years. It started at around £25,000 . . . it’s now £22,000 . . . it’s going to get reduced again, to probably nearer £20,000. Now, that’s been okay because I’ve been able to offset it with bids to County for Breakthrough money, for One to One money etc, which I’ve managed to use and channel and put into different areas. That’s going to dry up, so the only barrier I can see to doing what I’ve been doing in the last three years is financial. Um, and I think that’s why ‘S’ is so keen to, to, sort of, keep his links with the Science Learning Centres, because he feels that it is a really profitable way for him to train his staff, um, because, because that will be the biggest barrier for all training in the future.” WM/SMT1ab [312-327]

5.3.2. Organisation in schools The impact of CPD on schools in terms of missed teaching was discussed earlier as a significant factor influencing choice of the type (mode) and duration of courses on offer. The ‘cost’ to schools is not only financial but also, more importantly perhaps, in time and learning missed by classes. This is particularly sensitive when pupils are in examination years. Disruption is felt by the teachers because of having to set work for classes before they go on courses and then make up ground when they have returned to school. These in-school factors either limit numbers of teachers who can be 40

allowed out at any one time or mean that it is almost impossible to release teachers to attend CPD events. A senior manager in a London school said this: “Basically I work within school rules that were given to me by the business manager who runs cover. I’m allowed no more than two people out on any day . . . so sometimes no matter how brilliant a piece of training it is . . . it can’t happen because it doesn’t fit on the calendar and so money and dates and all those implications I don’t get to override them very often.” L/SMT1 [573-577]

Later, in the same interview, this senior manager talked about the ‘rarely cover’ rule and its impact on the school. The main point here is that protecting non-contact time for teachers means every class period missed for CPD is now more likely to be covered by supply teachers with a consequent impact on budgets. “So we’ve got no leverage left there at all and the impact (of the rarely cover rule) on our budget is absolutely phenomenal. Teachers are still being paid the same amount of money with the same amount of non-contact, so all that leeway I had from asking so and so to cover this or that, and that assumed goodwill has gone. I’m not saying I disagree, because I can imagine that being a main scale level teacher these days and knowing that you’re not going to have that little bit of non-contact nicked all the time is an absolute blessing. I don’t have a problem with it but the knock-on effect (on CPD) is huge and I think bigger than most people ever anticipated before it came in.” L/SMT1 [867-873]

The organisational and financial pressures on CPD in schools have meant a number of cases where managers are considering whether in-school, bespoke training might be easier to manage and finance but still provide high quality training rather than releasing teachers to attend courses provided elsewhere. In a case where CPD organised by the local SLC on enhancing science using ICT was provided at the school, a senior manager highlighted benefits of what he saw as the high quality of focussed training designed to meet the school’s needs. “. . . this (in-school bespoke CPD) is quality CPD, because I get an input in saying what I want and there’s no risk involved, which I think is really important to schools. When they’ve got a finite budget, we are able to say, this is my school, this is my staff . . . I don’t want to come to a course that you’re putting on that I might get two or three hours positive out of and the other is not relevant to my staff. The beauty of it, for me, is that you’re (the SLC providers) prepared to listen to what the school’s needs are. You will come in and deliver to the school’s needs and that, for me, is the best quality training you can get. It’s here, it’s in school, staff are comfortable doing it in school, they’re with their own colleagues and we can move them on as individuals. You know, I wish we could get more courses like that. Unfortunately, there’s an awful lot at risk with training providers that you’re not sure about . . . even at county level. You get people doing training and they don’t deliver what you ask for . . . and that, to me, is a crying shame because it’s a waste of time and, you know, but this to me seems the perfect way of arranging it, where you come in, you speak to the subject leader, you ask what they want and you create a package that fits the needs of the school.”

41

WM/SMT1ab [334-349]

The extent to which a course is properly matched to a school’s needs is important in the design and delivery of bespoke courses. In the course referred to above, it was noticeable that a teacher participant was much more critical of the match than senior management in the same school were. This degree of matching seems particularly important when ICT systems are involved as can be seen from this quote. “There was a problem between the wiki system we have in school and the one you (SLC trainers) showed us. It would have been more beneficial if you knew the school system and showed us how to make use of the wiki system we’ve got in school . . .” WM/T1 [213-217]

5.3.3. Location Careful thought was put into the location of SLCs when the NNSLCs was established so that the majority of schools in England are located at reasonable travelling distances. It is still the case, however, that even where SLC franchised courses are delivered at outreach locations (a particular effort was made to do this in the South West and North West), some schools find that location is still a significant problem affecting CPD choices. In these cases comments were from schools located on the coastal fringes of regions served by an SLC. In the case of the North West the problem of location of training seems to be driving the school towards school-located CPD. “HoD: In terms of people’s enthusiasm for going on courses or whatever . . . the closer to home it is, the more enthusiastic they are about going because I know for myself if I’ve got to go to a course in Manchester or Liverpool . . . before, you’ve got to go to Penrith then get a train from Penrith and then you’ve got to get a train or bus from there down to Didsbury or wherever, its 2, 3, 4 hours . . . totally crazy! INT:

A long day, isn’t’ it

HoD: So getting people to come to us, and delivering a half day course in little chunks I think is probably better than doing a full day somewhere else, because even if you come and do an afternoon you can usually, swap lessons around and free people up for an afternoon far more easily than you can for a whole day.” NW/HoD1 [170-181]

In the case of a teacher being released to attend a four part residential course in leadership at the NSLC, the Deputy Headteacher felt the course (leading to an M-level qualification) provided a link with HE that might otherwise not exist in the area. This additional sense of educational isolation led him to wonder if a training hub, involving the NSLC, could be established at the school. “One of the things I’d like to see happen is that there is a school in Scarborough, it would be nice if it is (ours) . . . becomes a hub centre for such activities, because we are such an isolated area. We are one of the most isolated areas in the country as far as

42

higher education is concerned. It will be nice if people did not have to travel and if people could come here, just come to a local school to do something at the Masters level.” NSLC/SMT1b [377-381]

5.4 The ‘routes model’ of CPD impact In our previous study of targeted CPD in science we proposed a model to describe and account for impact, the ‘classroom impact routes’ model (Bennett, Braund and Lubben, 2010, p. 15)

There are three routes to impact. The P1 (or personal) route is the most direct, affecting a participants’ own content knowledge, skills and professional knowledge that result in actions and changes with their own classes and pupils. In the P2 (or peer) route actions and outcomes resulting from CPD are disseminated to and affect the teaching of participants’ immediate peers. In the final P3 (or policy) route CPD outcomes might result in changes to department or school policy so that a wider number of teachers are affected. The analysis of cases in the current study supports the usefulness of this model in accounting for and understanding the impact of professional development. We found many of the outcomes attributed to the three routes in our previous study were also present in the cases studied here. What follows is a brief summary of examples of significant new insights on CPD impact in relation the routes and that have not already been discussed in this report.

5.4.1 The P1 (personal) route to CPD impact As in the previous study we found examples of P1 impact in all cases. Previously, we had not studied cases from leadership courses and so it was interesting to note that P1 outcomes still occur, with 43

highly experienced practitioners, even when intentions of participants and their managers for CPD were clearly focussed on managing macro changes and stimulating colleagues (i.e., action in P2 and P3 routes were expected most). One of the most experienced participants in the cases was a Head of Department using training on How Science Works (at an LEA outreach centre) to drive forward significant changes in teaching for his department. Even here there was evidence of a significant personal gain in the ways in which the CPD challenged the ways in which he had previously approached his teaching. “I was quite didactic as a teacher. I was quite, you know, ‘stand up look at me I am the font of all knowledge’ and I think that just by being challenged through the How Science Works course, it has drawn me away from that, which I know that I needed to do. It’s sort of given me ideas and different ways of thinking about things and looking at things from the pupils’ perspective a little bit better and I think that it’s challenged me in that way”. NW/T2 [201-206]

In one case of significant investment in leadership, a four part residential course at the NSLC, the teacher was firm that she mostly valued the experience for personal gain rather than having the skills and knowledge to manage change. However, as a result of the course she felt more prepared for leadership, which was what her managers had hoped for. This segment of her interview is quite revealing about the power of CPD to impact a teacher’s personal professional life. “Before I started the course, I expected the impact to be on myself and the class and possibly the department. I really did not think about the whole school. The course went way beyond any expectations I had. It was one of those moments that when I went on the course it really changed what I thought about CPD, how I felt about it and it really opened my eyes. I know it sounds a bit corny, but it was a life-changing moment.” NSLC/T1 [83-88]

In this case there was a gradual shift in emphasis from gains in the P1 route to gains in the P2 and P3 routes as the CPD proceeded, though the personal impact was never lost sight of. There was a feature in the personal, P1, route that we had not seen before. In one case we noted that the presenter’s style influenced the participant’s teaching. This was another case of an experienced teacher looking mainly for P2 and P3 gains but realising that his teaching had been affected in subtle ways through a process of gradual ‘osmosis’ of teaching technique and style into his personal teaching space. T2: “What I’d say is, if you’d go along to a session like that, and to an inset, and you see someone delivering something and then doing it confidently you can take that into your classroom teaching anyway, that sort of style, and you always pick up these things. INT: So even though it’s not about teaching though, you’re looking at the people, the way that people are presenting, so that’s interesting.

44

T2: Yes, you see people presenting something well organised and confidently and you sort of . . . all of this adds to your kind of . . . I like the way that person’s . . . I like the way that’s being done, so, but I can’t remember anything specific.” L/T2 [227 – 236 seq]

5.4.2 The P2 (peer) and P3 (policy) routes to CPD impact As might be expected largest gains on P2 and P3 routes were connected with leadership courses and examples have already been discussed. In leadership courses already having a management role, or being about to take one on, helped embed management skills and knowledge. It might also be the case that some P1 (personal) change is required to bring about effective changes on P2 and P3 routes. In the exchange below it seems the course helped a teacher to be more assertive so that delegation in the department might be more effective. SMT1: “He’s very nice, but not willing to sit by and let other people decide apparently that’s not their job when it clearly is . . . INT:

He’s being more assertive?

SMT1: Yes, quietly assertive.” L/SMT1 [726-734 seq]

There was some evidence that outcomes of courses in pedagogy or curriculum also have potential to bring about fundamental changes in the ways departments work. In the case quoted below, from a course on enhancing science using ICT, the Head of Department saw training as helping to bring together a previously disparate department founded on separate science disciplines: “I hope they would appreciate this idea of a collegiate approach, working together. We have now set up a teaching and learning challenge that we’ve used with other CPD that we’ve had since, and the Science Learning Centre . . . and kept us working as a team more and sharing ideas. Something that I felt when I first came, was there was a very discreet biology and chemistry and physics. Ah, the few members of staff who were good and outstanding weren’t necessarily passing on that knowledge to others. I’m not having a go at them, but that’s just what it is. I came . . . as a teacher I did exactly the same, but I’ve tried to change that and I think others have as well. So this has helped a lot.” WM/HoD1 [142-152]

As in our previous study, we found that participants from primary schools expected to at least contribute to changes in P2 and P3 routes if not to initiate changes themselves. There seems to be a strong sense of collegiate responsibility that drives primary-based CPD. This was strongly emphasised by this head teacher of a primary school: “The courses that I am looking for are the ones that address the school’s needs first and foremost, that’s what I will be looking for and secondly then it would be for the actual person themselves. Now if the two combine then that’s great, but I’m wanting value for money basically and sometimes it might just be to develop a person on an aspect

45

that they are weaker on but mainly I’m looking for something that can then feed back into the school to help us as a school keep moving forward and improving.” Y&H/HT1 [44-50]

5.5 Schools’ implementation space In our previous study we coined the phrase ‘implementation space’ for CPD. This describes ways in which the human and physical resources in a school are brought together to plan, manage and implement CPD and its outcomes. Thus it describes the management systems organised to identify individual, departmental and whole school needs for CPD and ways in which outcomes are facilitated, for example through dissemination events, provision of equipment or other financial support and support for teachers to collaborate, reflect on and plan for development or change. Implementation space is also occupied (or at least it should be) by processes to collect and analyse evidence of CPD impact. This takes place through observations, monitoring of teachers’ and departments’ action plans, pupil work monitoring, surveys of pupils’ voice and collection of pupil performance and other data. As we reported earlier, in this study, this part seemed poorly developed or it may have been that the study was not able to reveal evidence that remained obscured in complex cases of implementation spaces. We analysed the reports of the 14 cases for comments mentioning positive (facilitating) and negative (hindering) factors in schools’ implementation spaces. These follow. Numbers in parenthesis after each example show the number of cases in which comments were seen. Positive (facilitating) aspects of schools’ implementation spaces were:       

CPD is consonant with priorities for development (8) A culture of openness and sharing ideas (7) Matching staff with CPD: choosing staff most likely to bring about changes (4) CPD finances are flexible enough to meet needs identified by teachers (3) Having support staff to implement post-CPD developments (in the cases of an ICT course) (2) Time provided for colleagues to collaborate following CPD (3) Coaching arrangements in place to support development of teaching (2)

Negative (hindering) aspects of schools’ implementation spaces were:     

Lack of time to share and develop new approaches from CPD (3) Lack of resources to invest in equipment seen on courses (2) Organisational arrangements of the school: split site/ short lunchtimes (2) Resistance of some teachers to changes suggested by CPD (1) Incompatibility of school’s ICT systems (with that of the training) (1)

Finally it is interesting to note the comments made by this senior manager, identifying the aspects of her school’s implementation space for CPD. The comments reflect a wide and open view of CPD and its ramifications for the school. 46

“We expect all staff to appreciate that CPD takes very many forms. It could be research for instance, action based research in the classroom. It could be some reading. It could be some shadowing of other people, a whole variety of activities. We’ve had staff … brainstorming and coming up with a long list of things that they feel are valuable forms of CPD. So that they can recognise how many different ways that can happen.” “. . . It’s not just students that learn here. We all learn here, every single one of us, and we all have that responsibility as a professional, to keep learning and to keep developing and that’s very much the ethos that we’re trying to generate . . .” “We had a meeting on Monday where the principal talked to everybody about the ‘White Paper’ and the financial implications of what’s going on currently, curriculum implications – fantastic CPD. Everybody sat there and learned a lot about where we’re at now, why we’re there, what’s going to happen, what are the implications for us as a school and so on. A really, really good CPD. “In February we are taking our middle leaders and aspiring middle leaders away for a Friday night/Saturday session where we are looking at the whole concept of leaderships versus management. Then reviewing themselves as leaders. What sort of a leader are they? What sort of leader do they want to be? How are they going to get there? So there’s a lot of in-house CPD . . . “ “. . . I think the area . . . where we are lacking, and that we need to address currently, most urgently, is for our support staff, our admin support staff in particular. TA’s, teaching assistants, that programme is there . . .” NW/SMT [27-69]

47

6.0

Conclusions and Recommendations

6.1.

General comments

Combining the collection of quantitative survey data with qualitative case studies of enactment of CPD outcomes in schools provides fruitful material for analysis of impact. Survey information on CPD choice and expectations and outcomes, on its own, provides a general picture of the processes and impact of CPD related to mode and theme of courses. However, school-based factors for CPD impact remain highly situated and so individual, case study research is essential. The mode of a CPD course cannot be effectively separated from its theme and in many cases it has been problematic, in this study, to tie specific courses to particular modes. It has been possible to collect and analyse data across a wide variety of course types and school situations using a collaborative research model for multi-site research. This required extensive training and support for the personnel involved. The study and establishment of the collaborative research community provides a legacy that enhances capability for research and sharing research intelligence across the network. The mode of a course strongly influences schools’ choices for CPD. Teachers and managers often found it hard to produce systematic evidence of impacts of CPD on teachers’ professional learning and practice or on pupils’ learning, attitudes or behaviour. Bearing this in mind there are still plenty of examples of claimed impacts on all CPD outcome routes (P1, P2 and P3 of the ‘routes model’). Impact and challenge to existing ways of teaching was claimed for all modes and themes of CPD but there were far more cases for Mode C (multi contact/Leadership). Even where courses are clearly targeted at leadership, participants still show strong personal gains that change the ways they see their teaching. SLCs have advantages of being able to provide a high quality training environment, experts in CPD content of courses, a space allowing concentration away from the day-to-day life of school and the sharing of expertise with teachers from a wide variety of educational situations. Nevertheless, a number of school-based factors including cost, location, lower risk of situational relevance and perceptions of deep professional learning in a collegiate environment may persuade school managers and leaders to favour bespoke (Mode D) courses in the future.

6.2. Choice of and expectations for CPD 6.2.1. Choice of CPD Choice of CPD is influenced by a complex set of interacting factors; cost, location, duration, likely disruption to teaching, disruption to personal lives and the personal expectations of teachers and their managers. Case studies revealed that cost was a high priority affecting choice and that, without the impact/enthuse awards courses provided by the NNSLC, selection of courses might not even be considered. 48

The BOS survey showed choice is influenced by self–identified factors related most to the teacher (97% claimed needs are self identified). However, approximately 3 in 4 teachers still recognised the influence of their managers. This may vindicate the stated CPD policy of many schools in that they often made a case for a ‘bottom-up’ approach to CPD where teachers are required to self-assess CPD needs but where these are then moderated by overarching requirements of departments or schools to bring about macro changes in teaching or the curriculum. As in our previous study, we found strong evidence, from the case studies, that it was where this moderation process resulted in harmony between personal and institutional priorities for CPD that gains were likely to be most secure. BOS and case study data showed that the mode of a course strongly impacted the CPD choice process. In the BOS, 56% cited disruption to teaching as the most influential factor in choosing mode. Case studies revealed this was particularly sensitive where examination classes might be affected. One in three selected a course on the quality of the learning experience and here, personal or received knowledge of SLCs and the quality of what they provide is important as these quotes show: Well, we have had a lot of positive experiences with local Science Learning Centres … I was aware of some of the resources coming out of them and at my last school, in Yorkshire, I went to the NSLC at York and it was great … NW/T1 [79-98] I have a high regard for the NSLC … just the ethos and the impact of the building and what it stands for … NSLC/HoD1 [91-5]

Multi-contact (Mode C) courses were selected most often on the basis of thoroughness of the learning experience. Thoroughness was also associated with bespoke CPD which has the added advantage of being less disruptive to teaching. While there were a number of advantages to Mode D (bespoke) courses including cost, immediate relevance, rapid dissemination and collegiate involvement, some teachers and their managers seemed to prefer an environment for CPD away from the busy and sometimes distracting routines and duties in a school. Half-day courses seem hardly worth the cost and time for travel or setting of work compared with day courses. Twilight (after school) sessions are often seen as inappropriate and ineffective, due to teachers’ levels of fatigue. The ‘rarely cover rule’ leads some managers to see advantages of in-school bespoke modes of training. An advantage is that bespoke courses carry less risk that content is irrelevant or remote to schools’ needs or matches poorly with resource provision. This seems particularly important for courses with ICT or equipment-based elements. Location is important particularly for schools that are on the periphery of areas served by SLCs. In these cases support for bespoke or outreach provision at ‘training hubs’ is strong.

6.2.2. Expectations BOS results showed that more than three out of four teachers expected their CPD course to produce new teaching resources and strategies and that this was strongest for teachers attending singlecontact (Mode A) courses. Overall, just over one in three teachers expected the CPD experience to 49

improve their subject knowledge or answer queries regarding education policy or practice, but half of teachers expected to have some role in disseminating course outcomes. However, dissemination actions (action on P2 and P3 routes) were strongly associated with course mode. Participants in Mode C courses more often expected to focus on networking or resolving policy/practice issues. Teachers involved in Mode D courses more usually expected to gain exemplary teaching materials and strategies for dissemination to peers and develop teacher networks. Teachers taking CPD courses strengthening Pedagogy naturally expected to obtain novel teaching materials and approaches but fewer than average expected to be able to network with colleagues or respond to policy/practice issues. Although results of the BOS showed that expectations for networking or policy change were not particularly strong for those following curriculum courses, case studies revealed that these outcomes may depend more on the position and power of the teacher in a school’s implementation space to bring about change.

6.3.

Initiating and sustaining change in practice (including dissemination of CPD outcomes)

All course modes have similar capacity to challenge the ways in which teachers teach (that is, at Guskey level 2). BOS results showed this was true for about 2 in every 3 teachers, though only 1 in 6 were challenged very much, except for courses in mode C, where this was closer to a quarter of teachers. Leadership courses provided much more challenge than ones focussed on either curriculum or pedagogy where nearly all (94%) felt teaching was challenged a lot or to some extent. Case studies revealed that impact on own practice (the P1 route) was universal, even where teachers were highly experienced or already had significant management and training roles. Examples of personal (P1 route) changes were often subtle, sometimes peripheral to a course’s central intentions, or through casual ‘osmosis’ from practice experienced through CPD into one’s own pedagogy. As we saw in our previous study, changes in practice often have to be proven to be successful at a personal level before a teacher feels able to communicate and recommend them to others. There are overlaps between themes that make it hard to associate the level of impact on different routes with the theme of a course. For example a course focussed primarily on curriculum innovation or change could have just as much, if not more, potential impact on pedagogy. This was particularly noticeable in the case of a course on How Science Works. In case study interviews, the most commonly expressed advantages of multi-contact courses (Mode C) over other modes were that they provided cognitive space for deeper reflection on teaching and to test innovation and actions with wider impacts for teaching, such as assessment for learning. Gaps between days or blocks of CPD allowed deeper reflection on practice and for participants to clarify and debate effectiveness of changes in practice or policy with other teachers. Tasks set between blocks appeared to have facilitated and enhanced the extent of professional reflection. It was very rare for course outcomes not to be at least considered for dissemination (less than 6% of all respondents to the BOS). Most thought of CPD outcomes benefitting colleagues as well as themselves and pupils. About half disseminated outcomes through meetings, more so for Mode C courses, and a similar proportion informally (through conversations with colleagues). BOS results 50

showed that formal production of training materials happened in only 1 in 4 instances, most commonly in leadership courses. As might have been expected, reported benefits for whole departments or Key Stage teams and for whole schools were particularly prevalent for Mode C and Leadership courses. Mode A and Curriculum and Pedagogy courses seem to have benefitted a few teachers located in one school, while Mode D courses had potential to reach teachers in other schools, presumably when outcomes were shared in school clusters. Course materials, procedures and techniques were most often used within weeks of the CPD event, much more so in Mode C and leadership courses (15% more than other themes).

6.4.

Evidencing impact

6.4.1. The ability to provide robust evidence of impact More than half of respondents to the BOS said they were uncertain of impact, reflecting a similar trend in case studies where, in less than half of cases, was any substantial evidence of change provided. Where evidence was cited it was rarely from ‘hard’ sources such as surveys of pupil attitudes, test results or systematic observations of lessons. It seems that some school managers find it difficult to attribute impacts to CPD particularly where a teacher’s role has changed, for example having to take responsibility for a subject or phase (Key Stage). The most systematic, ‘hardest’, evidence of change was provided from one case in a Mode C, leadership course. Here there was evidence of changes in pupils’ behaviour, their science results, uptake of science subjects, pupils’ ability to work independently and teachers’ use of questioning techniques. In this case, the implementation space that facilitated such a systematic collection of evidence seemed due to a combination of efficient but responsive management of CPD and the nature of the course, that set tasks between blocks of contact demanding proof of impact on a number of different parameters (for example, not only requiring impact on pupils’ attitudes).

6.4.2. Impact on Pupils (Guskey’s level 5) The BOS findings showed that improvements in pupils’ science skills were most commonly reported even when courses had not had a specific focus on practical science or ‘How Science Works’. Improvements in pupils’ general skills (such as ‘thinking’) were reported more often for Mode C and Leadership courses, and less so for Mode D courses. An improvement in pupils’ attitudes to science was relatively more frequent for Mode C and Pedagogy courses. It was notable that changes due to courses intending to improve Pedagogy and Curriculum were, generally, less well evidenced probably because leadership courses (as shown in the case mentioned above) have more built in opportunities requiring reporting evidence of impact to course peers.

6.5.

Researching CPD impact

6.5.1. A community for collaborative research Two of the project’s four aims were concerned with improving research capacity across the NNSLC and providing a mechanism through which member centres could share practice in relation to the evaluation of their activities. The project was based on a model of research where a community of 51

researchers was constructed to take part in the collaborative collection and analysis of data. CELs, the key members of this community, came to the project having a wide range of backgrounds in educational research. Very few had extensive academic experience as researchers in education, a few had some previous contact through studying for a PhD or Masters degree or were registered for higher degree course and in early stages of research. The majority of CELs had little or no previous research experience. The team realised, at the early stage of forming our proposal, that research training would be required to ensure common understanding of methods and hence improve reliability of data collection and analysis. A second aim of training was to familiarise CELs with research methods and tools (such as the BOS) that they could use and adapt to carry out evaluation research at their own centres. While we did not set out to systematically collect data on the efficiency or impact of these efforts in collaborative research, there are some observations from having engaged in the process that might be helpful for the NNSLC and others wishing to use a similar research model. 1. The BOS proved an easy to use and effective tool for collecting and analysing data on teachers’ views on the CPD process, its outcomes and evidence of impact. All CELs were trained in designing a BOS survey instrument, analysed data for their centres and reported findings orally at the second workshop. The BOS is available to all centres via the NNSLC Network licence. A disadvantage of using BOS is that statistical analysis is at a fairly basic level but, as can be seen from this report, it is capable of providing sufficient findings for useful comparisons between courses and modes to be made. 2. Scrutiny of CELs’ reports showed that, in almost all cases, interview questions had been asked in a fruitful way using appropriate prompts and probes where necessary and with a degree of flexibility that allowed respondents to report freely what had taken place in their schools. 3. There were just three cases where interviewers pursued agendas, slightly away from the agreed line of questioning, that had some bias favouring a particular mode of course delivery. Interestingly, these slight diversions did not appear to have distracted or diverted respondents from the previous thrusts of their responses. 4. The allocation of examples of interviewees’ responses by the CELs to the themes and categories set out on the analysis framework was usually consistent. The training and careful explanations of the categories seemed to have paid off. The routes model of CPD outcomes seemed to have been understood though CELs had more problems in providing examples of schools’ implementation spaces.

6.5.2. Lines of evidence for CPD impact In this study we collected data along similar lines to our previous study. The main differences were that, in the first study, we collected data from in-depth interviews at two points, soon after CPD and at some time later (3-6 months from course completion). We also collected documentary evidence of teachers’ plans for CPD and written records of implementation/action provided by SLCs and/or schools. Thus, at the first stage interview, we were able to question teachers on what they had planned to do in school as a result of their courses and, at the second stage interview, whether these intentions had been carried out and, if not, why that was. In some cases teachers provided additional 52

documentation such as training resources or schemes of work and pupils’ materials that had been constructed or changed in the light of CPD. The research evidence here is rich enough to draw the conclusions we have made but we do recognise that the lack of evidence for impact that teachers were able to cite may have been connected with not being able to connect intended, planned actions with real evidence that anything happened in relation to these intentions. In future research, at whatever level, the link between intentions and actions relating to CPD is important to explore.

6.6.

Recommendations

Two sets of recommendations are made for SLCs/NSLC and the NNSLCs as a whole and one for schools (and colleges).

6.6.1. Recommendations to SLCs and the NSLC for the design and delivery of CPD Provision of CPD in all subjects and particular for STEM should be set against the national picture in a rapidly changing political and financial landscape. The quality of courses provided across the NNSLCs is rarely criticised, but the future of the NNSLCs cannot rest on past accolades. Given the current educational and financial climate and factors identified in this study that increasingly affect choice of CPD provision, the NNSLCs should seriously consider the extent to which it is able to meet an increasing desire for bespoke and off-site whole school delivery of CPD. The greater impact of Mode C and Leadership courses in the many examples highlighted in this report cannot be ignored and the NNSLCs should consider how it can increase provision here, particularly at all SLCs. In the planning, design, marketing and delivery of CPD courses, SLCs and the NSLC should also consider:  using the outcomes of this and other studies to make clear to potential ‘clients’ the values and advantages of different modes of CPD;  clarifying how content and outcomes can be better matched to individual client’s requirements. This is important for all modes not just as ‘a given’ for bespoke courses. This could be achieved by allocating more one-to-one contact between course tutors and teachers in face-to-face or virtual environments;  being sensitive to teachers’ perceptions of workloads and fatigue associated with half-day and twilight courses. Half day courses may still have value, particularly where they are part of school-located or bespoke provision;  that action planning and target setting processes, such as ‘the toolkit’, associated with CPD are foregrounded and allocated sufficient time in all courses, especially those having singlecontact;  setting up systems to maintain closer contact with teachers who have attended single event CPD and encouraging them to provide evidence of impact of CPD (Modes A and B);  ensuring that bespoke courses are ‘truly bespoke’, in that they are extensively negotiated with clients at the planning stage and take account of specific learning and resource situations of schools and colleges. This is particularly important where science equipment and/or ICT are involved; 53

 

recognising that most teachers completing CPD may wish to influence colleagues and providing guidance and advice on all courses as to how this might best be achieved; taking account of the remote locations of some centres from SLCs and the possibility of strengthening local ‘hub’ or ‘bespoke’ provision.

6.6.2. Recommendations to the NNSLCs for the evaluation of CPD impact In evaluating impact of CPD courses, the NNSLCs should consider: 

 

making use of the online survey facilities of BOS or similar systems (such as ‘survey monkey’) to collect large scale data from courses provided across their networks. The importance of collecting and sharing these data should be stressed to every participant at every course; exemplifying examples and methods through which CPD impact on teaching and on pupils’ performance, attitudes and behaviour can be collected, analysed and communicated; reporting data on CPD/course impact via local communications. This is already done nationally by the NSLC on behalf of the NNSLCs, but SLCs have an opportunity to do this more locally. Local communications could include case studies of schools where there have been successful examples of evidenced CPD impacts.

6.6.3. Recommendations for schools and colleges 

schools should use their existing monitoring and lesson observation systems more effectively to provide more robust and accountable evidence for classroom impact of CPD. This could include a requirement for teachers and departments to account for evidence of impact;



teachers should be encouraged to collect examples of evidence of impact of CPD. This might include surveys of staff use of resources, questionnaire data on pupils’ attitudes to new teaching or evidence of improved pupils’ performance in tests, examinations and homework or reduced incidents of disruptive or inappropriate behaviour where these are clearly relevant to the areas addressed by CPD; In the case of bespoke courses, schools should ensure that they enter into a clear dialogue with SLCs on requirements and adaptations of courses to meet the school or college’s needs.



54

7.0

References

Adey, P. (2004) The professional development of teachers: practice and theory. Boston, MA: Kluwer. Bennett, J., Braund, M., & Lubben, F. (2010). The Impact of Targeted Continuing Professional Development (CPD) On Teachers’ Professional Practice in Science. Main Report. York: University of York, Department of Education. Boyle, B., Lamprianou, I., & Boyle, T. (2005) A longitudinal study of teacher change: What makes professional development effective? Report of the second year of study. School Effectiveness and School Improvement, 16(1), 1-27. Boyle, B., While, D., & Boyle, T. (2004) A longitudinal study of teacher change: What makes professional development effective? Curriculum Journal, 15(1), 45-68. Cordingley, P., Bell, M., Isham, C., Evans, D., & Firth, A. (2007) What do specialists do in CPD programmes for which there is evidence of positive outcomes for pupils and teachers? Technical Report. Research Evidence in Education Library. London: EPPI-Centre, Social Science Research Unit, Institute of Education. Darling Hammond, L., Chung Wei, R., Andree, A., Richardson, N., & Orphanos, S. (2009) Professional learning in the Learning Profession. A Status Report on teacher development in the United States and Abroad. Stanford University. Available from http://www.asu.edu/ojs/article/view/147. Day, C., & Gu, Q. (2007) Variations in the conditions for teachers’ professional learning and development: Sustaining commitment and effectiveness over a career. Oxford Review of Education, 33(4), 423-443. Desimone, L., Porter, A., Garet, M., Yoon, K., & Birman, B. (2002) Effects of professional development on teachers’ instruction: Results from a three year longitudinal study. Educational Evaluation and Policy Analysis, 24(2), 81-112. Finegold, P. (2010). Professional Reflections. International perspectives on science teachers’ continuing professional development. York: The National Science Learning Centre and University of York Science Education Group. Garet, M., Porter, A., Desimone, L., Birman, B., & Yoon, K. S. (2001) What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38 (4), 915-945. Guskey, T. (2002) Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 45-51. Loxley, A., Johnston, K., Murchan, D., Fitzgerald, H., & Quinn, M. (2007) The role of whole-school contexts in shaping the experiences and outcomes associated with professional development. Journal of In-service Education, 33(3), 265-285. 55

Lydon, S., & King, C. (2009) Can a single professional development workshop cause change in the classroom? Professional Development in Education, 35(1), 65-82. National Science Learning Centre (NSLC) (2010a). Impact Report 2009-10. York: National Science Learning Centre, Downloaded on 13th March 2011 from: https://www.sciencelearningcentres.org.uk/research-and-impact/impact-reports National Science Learning Centre (2010b). Science Learning Centres 2010 2011 Secondary and Post16 Prospectus. York: National Science Learning Centre. Penuel, W., Fishman, B., Yamaguchi, R., & Gallagher, L. (2007) What makes professional development effective? Strategies that foster curriculum implementation. American Educational Research Journal, 44(4), 921-958. Sammons, P., Day, C., Kington, A., Gu., Q., Stobart, G., & Smees, R. (2007) Exploring variations in teachers’ work, lives and their effect on pupils: key findings and implications from a longitudinal mixed-methods study. British Educational Research Journal, 3(5), 681-701. Scott, P., Ametller, J., & Edwards, A. (2010). Impact of focussed CPD on teachers’ subject and pedagogical knowledge and students’ learning. Leeds: University of Leeds, Centre for Studies in Science and Mathematics Education (CSSME).

56

SECTION 8.0 Annexe A. Training schedules for the two research workshops at the NSLC First Research Workshop, 27th September – Schedule for the day

Times

ACTIVITIES: (Notes, resources/ materials)

10.30

Coffee

Registration, admin (Giving out folders to each CEL/Centre)

10.45

First Training Seminar (2 hours)

Introductions and schedule for the day and getting to know you. Introduction to the project and how it fits other work on CPD impact/evaluation. Group work – considering questions: Experiences of evaluation, how the project might add to personal professional knowledge? Training on the Bristol On-line Survey (BOS) system PART 1: Getting to know how it works and what it can do.

12.45

Lunch (30 mins)

Buffet lunch to be brought to training room and time to stretch legs etc

13.15

Second Seminar (1 hours)

BOS workshop PART 2: Participants will each go through a copy of the on-line survey we have devised and annotate making suggestions for improvements, e.g. changes in wording and additions and deletions of questions. Pairs compare notes and then give general feedback

14.15

Tea

In training room

14.30

Third Seminar (1 hours)

Introduction to case study work Rationale for case studies selection (chart with what we have selected provided) Individual and/or paired work to fill in the gaps for case studies e.g. for bespoke and other courses. What you will do next – revisiting the project timeline

15.30

End of workshop Admin.

Expenses claims, queries, etc

57

Second Research Workshop, 15th November – Schedule for the day Note Please bring your own memory stick with MoPD documents we have sent you for the workshop preloaded. Please also make sure you have your BOS login details.

Times 10.30

Action Coffee

ACTIVITIES: (Notes, resources/ materials) Arrival and registration.

10.45

First Training Seminar (2 hours)

Introductions and schedule/aims for the day. Training seminar 1 Individual CELs – report back on the pre-course task questions and observations of BOS data. Analysis using filtered BOS data: CELs given questions to ask of filtered data for modes and themes. Results can be stored online or onto your memory sticks.

12.45

Lunch (30 mins)

13.15

Second Seminar (1.5 hours)

Feedback and emerging trends. Buffet lunch to be brought to training room and time to stretch legs etc Training seminar 2: Review of interview procedures, protocols, and questions for teachers and senior/middle managers. Review of what analysis needs to be done. Each CEL will be provided with the blank ‘analysis framework’ for a participant case study and selected materials from transcripts of three persons: teacher, SMT and HoD (each one on different coloured paper). Data transfer and draft comments entered into blank frame.

14.45

Tea

15.00

15.30

Feedback from the activity In training room Timetable and admin for what to do from now until deadline for research reports to be sent to the central team. Individual planning (timeline activity)

End of workshop Admin.

queries, etc

58

Annexe B. Questions used in the Bristol Online Survey Instrument

59

60

61

62

63

Annexe C: 1.

Interview schedules used for the case studies

Interview Schedule for Course (teacher) Participants (RESEARCHERS’ VERSION)

Thank you very much for agreeing to talk to me about the impact of a course you attended at one of the Science Learning Centres. I have been asked by the Science Learning Centre Network to construct case studies of the impact of different types and content of CPD courses on classroom activities in a variety of school contexts. The discussion, as part of such a case study, will be confidential and the reporting will be anonymous, so I hope you can be as frank as possible. The interview questions are detailed below. NOTE: Letters and numbers in bold italicised text show how questions link with fields of the analysis framework (Annexe D). 1. The context: I would like you to talk briefly about your background in the school 1.1 1.2 1.3 1.4

How long have you been teaching and how long at this school? What are your specific responsibilities in the area of science teaching? Can you briefly summarise your wider role(s) within your school e.g., pastoral, year head etc? What are three characteristics of your school that are most relevant in helping you to achieve classroom impact of the course? (C4/D4)

2. 2.1

Expectations of CPD participation How did you decide to take this CPD course? What did you view as the major attractions of the course? (A1/C1) When deciding on a CPD activity, how important is the CPD mode (for instance a one day course, versus a several days block, or a sequence of inputs, a bespoke course at school, etc.) (A1)

2.2

We could consider CPD potentially to contribute to changes in three areas:  yourself and your classes,  your colleagues and their classes  your whole department or the whole school. 2.3

Which of these target groups did you expect to be affected most by your CPD? Can you give me some examples of ideas, concepts or skills you wanted to get clarified in your own mind? (B1-3) 2.4.1 To what extent were these aspects addressed by the course? What course activities helped? (A1, C2, D2) 3.0 Impact I would like you to talk about the impact the course has had (e.g. on your or your colleagues teaching and on your pupils). Can you tell me: 3.1 3.2 3.3 3.4 3.5

What are you doing now in your teaching differently than you did before as a result of the course (B1) Why have you changed these things and not others? (B1) What impact have these changes had on your pupils? (B1) What evidence do you have that the changes have been a success (e.g. in terms of pupils’ learning and attitudes)? (B1) What about your colleagues? Have they introduced any changes as a result of the course or ideas and materials from it? How do you know about what has been the effects of any changes in what they have done? (B2) 64

3.6 3.7 3.8 3.9

What about changes in department or school policy? Have there been any marked shifts in the ways the department or the school now approaches x or y (B3) What are the factors that have helped you implement changes as a result of the course (e.g. people, school CPD and feedback structures of the school, resources, time and so on)(C4) What are factors that have hindered implementation of changes as a result of the course (e.g. people, school CPD and feedback structures of the school, resources, time and so on)(D4) How would the impact have been different if the CPD had been structured differently such as a one-day course, on several successive days, as a sequence of CPD contacts with tasks between them or CPD organised and delivered specifically for your school or organisation(bespoke) (change according to mode of CPD of the actual course experienced) (A1)

4.0 Further plans for development/change I would like you to talk about the influence the course has had on any further plans for development. Can you tell me: 4.1. What more do you plan to do in your own teaching as a result of the course over the next six months – year? 4.2 Have you any further plans to disseminate ideas/materials to other members of staff? If so please tell me about these. (B2-3) 5.0 5.1

Any other comments Do you want to comment on any other aspects of potential impact of the CPD programme?

Thank you very much for taking the time to talk with me

65

2.

Interview schedule for PD Managers, SMT or HoS/HoD RESEARCHERS’ VERSION

Thank you very much for agreeing to talk to me about the impact of the ….. course at the ……….. Science Learning Centre. I have been asked by the Science Learning Centre Network to construct case studies of the impact of different types and content of CPD courses on classroom activities in a variety of school contexts. The discussion with you, as part of such a case study, will be confidential and the reporting will be anonymous, both inside and outside your school, so I hope you can be as frank as possible. The interview questions are detailed below. NOTE: Letters and numbers in bold italicised text show how questions link with fields of the analysis framework (Annexe D). 1. 1.1 1.2 1.3

Context – Your role Could you briefly explain your management role(s) within the school as far as CPD is concerned? How long have you been in this role, and how long at this school? Can you briefly describe the school’s or Department’s policy on CPD? (C4/D4)

2. Expectations of CPD participation One could consider CPD potentially to contribute to changes in three areas:  The teacher participant and her/his classes,  The participant’s colleagues and their classes  The whole department or the whole school. 2.1 2.2 2.3 2.4 2.5 2.6 2.7 3.0 3.1 3.2 3.3

3.4 3.5 4. 4.1. 4.2. 4.3.

Which of these areas did you hope the course participant would address most? (A2, B1-3) What were the school’s reasons for investing time (and possibly money) in training in XXXX (topic of the course)? (A2) How did you become aware of this CPD need? (C4/D4) Why did you opt for this particular mode of CPD course (One day, more than one day, sequence or bespoke). Have you considered addressing this need in any other way? (A1) What were the reasons for asking Mr/Ms [YYY] to go on this course? What personal qualities (of the participant) were involved? (A2) How did you expect [YYY] to change his/her classroom strategies? (B1) How did you expect [YYY] to share the CPD experience with colleagues? (B2/3) Impact of the CPD Are you aware of any impact of the course on ….. teaching? If so, how do you know there has been impact (i.e. what evidence, measures and so on?) (B1-3) Are you aware of any impact of the course on …..pupils? If so, how do you know there has been impact (i.e. what evidence, measures and so on?) (B1-3) Has there been any impact on the practice of ……..’ s colleagues? (in terms of their teaching, what they now do, their pupils and so on). If so how do you know there has been impact (i.e. what evidence, measures and so on?) (B2) What will……… do now as a result of being on the course (any further action? If so how will you monitor/manage etc? If nothing further why not?) What now for ………. ? (any ideas for his/her further PD) School’s implementation space How was the school able to support any changes in ……’s practice? Which three characteristics of your school may be barriers to achieving impact on classroom practice? (C4/D4) How was the school able to support YYY in sharing the CPD experiences? (C4) Which three characteristics of your school may be barriers to whole-school uptake? (D4) 66

5. 5.1.

Any other comments Do you want to comment on any other aspects of potential impact of the CPD programme on your school? Thanks for talking to me

67

Annexe D. The case study analysis framework provided for CELs Analysis framework for each participant in MoPD case studies

Analysis component

Nature of classroom impact

Evidence

Course-Mode characteristics

Participant’s characteristics

P1 implementation (on own classroom)

P2 impact (on participant’s peers)

P2 impact (on changing departmental or school policy)

(A1) Course-mode characteristics (B1) Examples of P1 impact (B2) Examples of P2 Impact (B3) Examples P3 Impact

(C) Factors that facilitated classroom impact of the CPD, relating to: (C1)The teacher’s pre-course expectations (C2) The characteristics and content of the course (C3)The personal characteristics of the teacher (C4) The school’s ‘implementation space’ for CPD 68

(D) Factors that hindered classroom impact of the CPD, relating to: (D1) The teacher’s pre-course expectations (D2) The characteristics and content of the course (D3) The personal characteristics of the teacher (D4) The school’s ‘implementation space’ for CPD Guidance notes for CELs The analysis components in the table summarise evidence of impact drawn from transcripts of interviews with course participants, SMT members and middle managers. The table will contain just short phrases against ‘source codes’ (from interview transcripts) summarising most important evidence. In the table and the headings below it, a letter and number reference (A1…, B1…, C1…, D1…) has been used to link items of the analysis framework to specific questions used in the interview schedules. You will see these letter/number references against the questions in each schedule. You will provide a commentary illustrated by suitable quotes (including ‘source codes’) under the sub-headings of routes of impact (PI, P2 and P3 – see below) and factors that facilitated and hindered impact of the CPD. Paragraphs under each sub- heading will vary in length, but a report for each participant should amount to no more that 10 sides of A4. The nature of impact column will contain phrases summarising actions/outcomes (see previous case study reports). The evidence column will contain phrases summarising what evidence is claimed or has been seen in terms of actions and outcomes. For example, evidence might come directly from sources suggested by interviewees’ such as marking, tests, pupil surveys, observation of teachers’ lessons my SMT or from more informal claims of pupils’ improvement through conversations, anecdotal notes of pupils’ attitudes and behaviour and so on. There might also be documentary evidence from schemes of work and lesson plans, instruction manuals and training materials for staff, resources added to VLEs and records of meetings of intra or inter-school events. Course or mode characteristics will be evidence that the content, style or organisation of the CPD had an impact on what happened or failed to happen as a result of it. For example a teacher might have expected the course to have addressed a conflict in their own perception of a teaching method or policy issue but in fact they were ‘none the wiser’ after the CPD. On the other hand a teacher might have expected to have picked up pedagogical content knowledge in the form of new approaches in teaching but in fact gained most in their own subject knowledge of an area to teach and so growth in subject confidence was an important outcome. The style and organisation (mode) of a course lies at the heart of this evaluation and here we might pick up comments such as; that a two or multi-part course, with set tasks between centre contacts, enabled participants to reflect on practice and plan and bring about significant change in practice or policy. A one-day course might bring about immediacy of action without a need to wait for a second tranche of training. This is likely to be the place where data answering our research question 3, about how schools approach their planning of professional development, given the choices, features Participant’s characteristics deals with the status, position or driving characteristics of the teacher participant within the CPD implementation process. Evidence might come from insights gained from 69

interviewing the teacher, but more often it is from comments made by middle managers and SMT. For example, a teacher might be thought of as a ‘mover and shaker’, a key player able to drive forward change by influencing other colleagues. Alternatively a teacher might be thought of as previously shy of CPD and in need of building confidence to move into a new area of teaching or with pupils of different backgrounds or abilities. The routes to classroom impact of classroom impact of CPD. In our previous evaluation of courses provided by the NNSLC we proposed a model of three classroom impact (CI) routes. Firstly, a CPD experience may change the participant’s knowledge, attitudes, skills or motivation who in turn implements these changes in her/his own classroom. Since this is the most direct trajectory from CPD to the classroom, this is labelled the P1 (or Personal) route. A second route for classroom impact, the P2 (or Peer) route, involves the participant sharing and adapting the changes suggested by the CPD with peers, which will then impact on the activities in peers’ classrooms. Lastly, on the instigation of the participant, CPD-based changes may be translated into changes in school (or departmental) policy, which in turn will need to be implemented by the collective school (or departmental) staff before impacting on classroom activities across the school (department). Since this is the most elaborate route for classroom impact, we have labelled this as the P3 (or Policy) route. The school’s implementation space for CPD. In the analysis of the case studies we coined the phrase ‘implementation space’. We found this a useful construct to conceptualise and rationalise complex school-based factors that affect the implementation of CPD outcomes. Implementation space encompasses the physical and human resources available in a school and the ‘macro’ strategic management processes and policies of the school regarding CPD that in turn impact ‘micro’ structural considerations such as timetabling and facilitation of CPD. It appeared that secondary schools have a more complex implementation space for CPD than primary schools. For example in primary schools CPD actions were the responsibility of the Headteacher in direct negotiation with course participants. In secondary schools there were several layers of responsibility usually involving a coordinator for CPD (often a deputy or assistant Headteacher), a line manager responsible for science and/or other faculties and a head of subject and/or a head of department (science) or of a faculty (of STEM subjects) - sometimes both. There is thus more likelihood in secondary schools, if layers of management are not fully coherent and coordinated by rapid and effective cross communication between them, for CPD intentions to remain unfulfilled or partially completed. There is also the possibility that intentions, expectations and outcomes of personal (P1) and peer (P2) CPD are not in sympathy and so conflict with whole school intentions and actions for CPD

70