Background

This review - What do teacher impact data tell us about collaborative CPD? - aimed to test and build upon the first and second CPD reviews and, in doing so, to explore the methodological issues related to studies that evaluate the impact on pupils and teachers, as compared with those which only evaluate the impact on teachers.

In the first review, we sought to identify processes involved in collaborative CPD interventions that have a positive impact on teaching and learning. In the second review, we systematically reviewed and synthesised the data from studies of individually oriented CPD, before comparing individually oriented CPD with collaborative CPD. In doing this, the searches for the first review were updated and the findings from the studies of collaborative CPD in the first review were applied to any additional studies of collaborative CPD identified in the second review. This enabled us to refine and build on definitions of collaboration established in the first review and to analyse in detail the nature and relative importance of collaboration, as there had been a significant growth in both activity and research in this field since the first review. Indeed, in the UK this seems to have been partly as a result of the first review.

The aim of this third review was to identify those studies of collaborative CPD which focused on teachers across the 5-16 age range, but which only provided data about teacher outcomes, to enable us to identify the impact and specific characteristics of teacher-focused studies. In a second stage of the analysis, we compared the processes and outcomes of the CPD described in them with those from the teacher-and-pupil focused studies that were investigated in our first and second reviews. We were interested in reviewing studies that explored teacher impact only to see what additional light they might throw on the first two reviews, in relation to aims, CPD processes, methods and findings.

Aims

In summary, our aim was to review systematically the literature investigating the impact of collaborative CPD that measured teacher-only impact, and then to compare two distinct clusters of CPD - studies that included evidence of the impact of the CPD on teachers and pupils (reviewed in the first and second reviews by this group), and CPD that focused only on teacher impact.

As with the earlier reviews, the Review Group hopes to make some of this evidence available to practitioners in an accessible and meaningful way, to highlight the areas in which further research would make a valuable contribution to CPD strategies, and to enable evidence-informed reflections upon implications with policy-makers.

Definitions

For consistency, we continued to use the definition of CPD we adopted for the first and second reviews:

Professional development consists of all natural learning experiences and those conscious and planned activities which are intended to be of direct or indirect benefit to the individual, group or school and which contribute through these, to the quality of education in the classroom. It is the process by which, alone and with others, teachers review, renew and extend their commitment as change agents to the moral purposes of teaching; and by which they acquire and develop critically the knowledge, skills and emotional intelligence essential to good professional thinking, planning and practice with children, young people and colleagues through each phase of their teaching lives. (Day, 1999, p 4

For the purposes of this review, 'collaborative CPD' refers to programmes in which there were specific plans to encourage and enable shared learning and support between at least two teacher colleagues on a sustained basis. 'Sustained CPD' refers to programmes that were designed to continue for at least 12 weeks. The review includes those studies of CPD which reported evidence of impact, either positive or negative, on teaching.

Review questions

The overarching question for the third review is as follows:

What can we learn from studies of sustained, collaborative CPD which set out to explore the impact on teachers and teaching but do not also consider the impact on pupils in the context of the evidence from previously data-extracted studies of collaborative CPD that consider the impact on both?

(For brevity this is sometimes shortened to: What do teacher impact data tell us about collaborative CPD?)

The evidence of the impact of collaborative CPD from studies that measure both teacher and pupil outcomes is taken from the first two reviews of CPD by this Group. The studies of collaborative CPD that measure teacher-only impact are identified from the searches of the first two reviews and the data are synthesised in this, the third, review. Comparisons are then made between the two clusters of studies - teacher-and-pupil focused studies and teacher-oriented studies - drawing on the syntheses in the three reviews.

Sub-questions

The first phase of the data analysis synthesises the data from studies that only measure teacher impact relating to the question:

What is the impact of sustained, collaborative CPD on teachers and teaching?

We then go on to look across studies of collaborative CPD from all three reviews and compare the nature of the studies in the two clusters - teacher-and-pupil focused studies and teacher-oriented studies. The comparison is structured around the following sub-questions:

Do the studies of the three different reviews provide evidence about different types of aims for the CPD, depending on whether they explore only the impact on teachers and teaching, or explore the impact on teachers, teaching and pupils?

Do the studies of the three different reviews provide evidence about different types of CPD processes and activities, depending on whether they explore only the impact on teachers and teaching, or explore the impact on teachers, teaching and pupils?

Do the studies from the three different reviews provide evidence about different types of outcome for the CPD, depending upon whether they explore only the impact on teachers and teaching, or explore the impact on teachers, teaching and pupils?

Finally, we explore whether studies that investigate sustained collaborative CPD use different study designs, depending on whether they explore only the impact on teachers and teaching, or explore the impact on teachers, teaching and pupils.

Methods

Identifying, describing and appraising studies

For practical reasons, the review focused on studies published after 1991 that were reported in English, although no geographical limits were set. We wanted to engage both primary and secondary practitioners, so the review includes studies that involved teachers of the 5-16 age group. The studies had to have a focus on teaching and learning, and outline the explicit learning objectives of the CPD.

All the studies included in the third review were identified through the searching and screening processes of the first two reviews.

The following methods were used to identify studies for the systematic map and in-depth review:

• a systematic search of the literature, using electronic databases, handsearching key journals, word of mouth, citations and websites
• the application of a set of initial inclusion criteria to the titles and abstracts thus uncovered
• retrieval of full reports, to which the criteria were re-applied to see if they were suitable for inclusion in the mapping stage of the review
• application of keywords to all the included reports using EPPI Centre core keywords, such as type of study, type of setting, age and curriculum focus, as well as a number of review-specific keywords to distinguish finer detail between types of intervention, teachers and processes
• application of a second, narrower set of inclusion criteria to the keyworded reports to identify studies that did and did not include student data
• use of EPPI Centre data-extraction software to extract data from the studies and to assess the weight of evidence they provided for answering the review-specific question
• extraction of data and quality assessment by two people operating independently and then reconciling any difference
• use of the evidence from the tables from the data extraction and quality assessment as the basis for synthesising the studies to answer the review questions

Results

Mapping of all included studies

During our first and second reviews, we sifted 18,963 titles and abstracts systematically, and reviewed 489 full text studies, 45 of which were identified as meeting the inclusion criteria for the current review and were included in a systematic map of the literature. The 45 studies were made up of 31 studies which contained teacher and student impact data (included in the first two in-depth reviews (N=17 and N=14, respectively)) and 14 studies which had been excluded from the in-depth reviews of the first and second reviews because the study reported teacher-only impact data and did not examine the outcomes of the CPD in relation to students.

Characteristics of all included studies

The majority of the 45 studies in the systematic map came from the USA. The educational settings in which the studies took place were predominately primary (N=29) and secondary (N=24) schools, while some covered both. The vast majority of the studies (N=42) focused on teaching and learning. The next most popular focus of all studies was curriculum (N=33), where the subject was likely to be mathematics, literacy (first language) or science.

Weight of evidence (WoE)

Of the 45 studies in the systematic map, 31 had already been reviewed in-depth in earlier reviews. Therefore the current review involved data extraction of 14 studies that measured teacher-only data. Of these 14 studies, three were judged to have low weight of evidence (WoE). As a consequence, the three studies were data-extracted, but were not included in the synthesis. Two of the resulting 11 studies were found to have high WoE and the rest were assessed as medium.

Findings and implications

Synthesis of findings

The first stage of the synthesis brings together the findings from the 11 higher WoE studies that only measured teacher outcomes. The syntheses of the studies that measured both teacher-and-pupil outcomes are included in the previous reviews by this group.

Types of study

In the majority of the studies that collected teacher-only data (7 out of 11) the research aims primarily related to the evaluation of a particular CPD design or approach in the context of a curriculum-based goal. In four cases, the CPD studies were directed mainly at the improvement of a particular aspect of the curriculum or teaching strategies, using the CPD as the vehicle for improvement.

In all cases but one the researchers provide data about the interventions which offered us the opportunity to:

1. identify and report on the CPD processes and activities for this group of studies
2. compare these across the two groups of studies: i.e. those which presented teacher impact data (synthesised in the third review) and those which also presented student impact data (synthesised in the first and second reviews).

Impact of the CPD in the teacher-only studies

We have categorised all outcomes in two broad clusters: behavioural and affective.

Impact on teacher behaviour:

• Teaching: In all but one of the studies, the teachers involved in the CPD interventions changed or substantially developed aspects of their teaching following the CPD intervention. The remaining study reported how the CPD helped embed professional collaboration among the teachers as an ongoing approach to professional practice, but reported no detailed data about the practical impact of this on teaching behaviours.
• Ongoing collaborative working: The studies all suggested that collaborative CPD processes were linked with a disposition to work and reflect collaboratively with colleagues as an ongoing process, whether or not this is an aim of the CPD.

Affective impact

All the studies reported both observable and self-reported enhancement in at least one of the affective aspects of professional learning:

• motivation
• confidence
• attitudes and beliefs.

CPD processes and characteristics in the teacher-only data studies

The evidence from this group of studies reinforced the findings about the nature of effective collaborative CPD from our previous CPD reviews. Specifically, the studies provided evidence about the positive benefits of CPD that:

• made use of peer support
• made explicit use of specialist expertise
• made explicit mention of involving the teachers in applying and refining new knowledge and skills and experimenting with ways of integrating them in their day-to-day practice (six studies involved action research)
• involved consultation with the teachers about their own starting points, the focus of the CPD, the pace of the CPD or the scope of the CPD
• involved teachers observing one another as an integral part of the CPD
• involved specialists in observation and reflection (as part of the CPD rather than exclusively focused on data collection).

Nature of collaboration in the teacher-only data studies

In the second review, as well as looking at the impact and characteristics of collaborative CPD, we began to explore the nature of collaboration in more detail and, in the light of the evidence from this review, advanced some tentative hypotheses about the characteristics of effective collaboration. These were as follows:

• Classroom-based activities may be a helpful factor in increasing the effectiveness of the CPD.
• Collaboration between teachers which is coupled with active experimentation may be more effective in changing practice than reflection and discussion about practice alone.
• Collaboration may be an effective vehicle for securing teacher commitment and ownership of CPD in cases where it is not possible for the teachers to select a CPD focus of their choice.
• Paired or small-group collaboration may have a greater impact on CPD outcomes than larger groups.

We applied the hypotheses developed in the second review to the teacher-only data studies synthesised in the third review. Doing so identified the following patterns:

• Location: The majority of the interventions took place, wholly or to a significant extent, within the teachers' own schools. In general, this finding is consistent with the proposition that CPD can be effective when it has a significant in-school component.
• Experimentation versus reflection: The majority of the studies combined reflection and discussion about practice with active experimentation in classroom practice. This is consistent with the trend towards paired collaboration and with the hypothesis that active experimentation may be an effective means in changing practice.
• Groupings: Teachers working in pairs was the most common form of collaboration, although the unit of collaboration was unclear in two of the studies. In some of the larger studies, there were opportunities for collaboration in larger groups as well.
• Voluntarism: In all but two of the studies, teachers were voluntary participants in the CPD intervention. However, it seems that the affective impact of collaborative CPD, together with the acquisition of new knowledge and understanding, engendered a sense of ownership of the intervention among teachers in all cases.
• Student orientation: While we did not explicitly aim or expect to find details about student outcomes in this third review, we did expect that teacher perceptions about the impact of the CPD on their students would feature within the teacher data. As this kind of teacher perception data was only sparsely reported, this did not prove to be the case.

In relation to the nature of collaboration, this review adds to our understanding of the nature of effective collaboration to the point where we feel more confident about our four propositions from our second review. In exploring the components of CPD that are linked to positive outcomes, we are noting strong patterns of connection rather than causation. Without further research in which the components are treated as independent variables, causation cannot be established.

How did the teacher studies compare with those reporting pupil data?

In the second phase of the analysis, we compared the nature of the higher WoE studies reporting teacher-only data (N=11 synthesised in the third review) with those providing evidence about impact on students (N=15 synthesised in the first review; N=11 synthesised in the second review; total N=26). Specifically we explored the four areas:

• aims
• nature of the interventions
• outcomes
• study design.

While reviewing the studies in depth, we noticed potentially interesting and useful patterns in the literature base of the studies. This was not written into our protocol but, as we feel that it sheds light on the aims of the study, we have included the comparison.

• Literature base: There appear to be two broad groups of research here. One group, which focused on changes in teaching and learning generally, treated CPD as one of several interesting variables, while the other generally functioned more as a set of evaluations of CPD in terms of teacher change. In this latter group, the teaching and learning processes featured much less prominently than the CPD processes. The CPD programmes, in which pupil and teacher data were collected, paid more attention to pre-existing evidence about teaching and learning than those where teacher-only data were collected. Since we do not know, in the teacher-only data studies, what the impact on pupils was, the lack of attention to the pedagogic research base may or may not be an important aspect of the study. It may be that the teacher-only studies did refer to pedagogic literature but did not have room to report it in the article. However, since they give considerable space to reporting the CPD literature, this seems unlikely.
• Aim: Teacher-only and teacher-and-pupil studies gave equal attention in their aims to exploring the impact of a specific teacher development programme or assessing the impact on teaching and learning of introducing specific pedagogic strategies (half of each group in each case). Studies which focused on teacher impact only were more likely to have an explicit intention to develop teachers' knowledge, understanding or skills, and were much more likely to have an explicit aim to change teachers' beliefs or attitudes. Studies which provide pupil and teacher data were inevitably more focused on improving pupil outcomes and, perhaps as a result, on enhancing teacher practice.
• Nature of the interventions: The key strategies used in the two groups of studies were similar in relation to:
  • the use of specialist expertise
  • creating opportunities for teachers to observe others teach
  • peer support
  • the use of workshops and seminars.

  All these strategies feature prominently in both clusters of studies, but there is a greater explicit emphasis on processes described as action research in teacher-only studies. In both groups of studies, however, programmes that made explicit reference to action research were very similar in content to those that described themselves as coaching programmes.

• Outcomes: All the teacher-only data studies focused upon affective outcomes, compared with fewer than half of the studies reporting student impact data. Changes in teacher behaviour was an explicit outcome of the vast majority of studies with similar proportions in each cluster providing evidence for this.
• Study design: All the studies were evaluations. The majority of studies in both the teacher and the teacher-and-pupil clusters were researcher-manipulated evaluations. The rest were naturally occurring evaluations. Control or comparison groups featured much more often in the teacher-and-pupil impact studies than they did among the teacher-impact studies. Teacher-only studies were much more likely to have collected data during the study than were those reporting student outcomes. Studies designed to explore the impact of CPD on teachers only were generally carried out over a longer period of time than those studies which also collected pupil data.

Nature of the studies

We had wondered whether the teacher-only data studies would provide evidence about teachers' perceptions of impact upon students. In fact, very few data about teachers' judgement of impact upon students were recorded. The implication seems to be that CPD explored by studies that focused on teacher-only data was aimed more explicitly at changes such as teacher knowledge, beliefs and understanding which cannot be directly observed. It could be that teacher-only studies set out to provide evidence about these phenomena as a proxy for direct pupil data. On the other hand, the studies which provided data on changes in pupil learning may have felt that this was sufficient evidence to imply changes in teacher attitude. Consequently, the teacher-and-pupil studies focused much less on affective outcomes.

Two broad areas of research emerged from analysis of the studies: there was either a focus on changing teaching and learning generally, in which CPD was treated as an incidental variable; or the studies set out to evaluate CPD and placed less emphasis on teaching and learning.

Furthermore, the CPD programmes in which pupil-and-teacher data were collected built more directly than the teacher-only studies on pre-existing evidence about teaching and learning. The CPD programmes in the teacher-only studies, on the other hand, focused more than the comparison group on pre-existing evidence about CPD. The teacher-only studies were, in effect, evaluations of CPD, while the other group of studies were explorations of effective ways of achieving change and improvement in teaching and learning.

There is a certain lack of potentially useful detail in reports of studies in both groups: the teacher-and-pupil data studies provide little information on the nature of intervention and underpinning rationale, whereas the teacher-only studies were lacking in evidence on teaching and learning.

Strengths and limitations

Strengths

One strength of this review is the way in which it builds systematically and cumulatively on previous reviews. In doing so it has continued to probe the questions raised in previous reviews about the emphasis on impact and the exclusion of other types of evidence. Another strength is the way in which the review grows from live concerns and consultation with policy-makers and practitioners through the involvement of a number of user groups in setting and refining the questions, and interpreting and disseminating the findings.

In particular, the CPD Review Group considers that the review has continued to help in the following ways:

• developing a taxonomy of collaboration which is meaningful and applicable to practitioners and policy-makers
• adding to the base from which we can continue to unpack the specific processes involved in the CPD intervention and identify those which appear to influence change in teachers' practice
• exploring further the effect and influence which external and specialist expertise brings to the design and impact of CPD processes
• identifying the patterns of research related to CPD and the relative strengths and weaknesses of studies that do and do not collect pupil outcomes data.

Limitations

• One limitation of the review is that we did not run any additional searches to see whether there were any other articles or reports covering student data for these programmes by the authors of the teacher-only studies, although the descriptions of methods and approaches within the articles that suggest this is unlikely to be the case.
• We were conscious throughout of the limitations of the data provided in the reports of studies we retrieved with regard to answering our review question. None of the studies was designed to answer our review question directly.
• In particular, we noted problems arising from the compressed timetable for completing the review. There were difficulties in responding to possible trends or patterns arising out of answers to the questions in our protocol by creating further tables. We were unable to go back to the original studies from the earlier reviews in the detail that we would have liked in order to follow up new points arising from the current review. For example, we would have preferred to carry out a more detailed analysis of the outcomes for teachers.
• We also noted the following in the individual studies:
  • There was a varying amount of detail about the sample in some of the studies, and some reviewers noted that they would have liked to have been given more detail about the sample background(s) in order to make the connections between contexts.
  • The different aims and foci of the studies lacked detail and, in some cases, clarity.
  • The overwhelming majority of studies were conducted in the USA and so it is not known whether the findings could also apply in other countries.
  • There may well have been additional fruitful data in a number of PhD theses and other studies. However, we were unable to retrieve these within our timescale and note that these data remain unexplored.
  • In some reports of studies, there is a lack of discussion of the effect of using the researchers as part of the CPD intervention on the evidence.
  • The small-scale nature of some of the studies included in the in-depth analysis.

Implications

Implications for practitioners

The research suggests that collaborative CPD is linked with positive outcomes regarding teachers' attitudes to working and reflecting collaboratively with colleagues on a sustained basis. Schools and CPD co-ordinators working with colleagues who have little or no inclination to work with others, should consider creating and resourcing opportunities for teachers to participate in CPD in partnership with one or more colleagues.

In cases where teachers did not volunteer to take part in the CPD but were required to do so, the collaboration designed into the intervention helped to convert initial co-operation into genuine collaboration. Schools and CPD leaders should consider paying attention to the potential benefits of collaboration when trying to meet the needs of disaffected or demotivated colleagues. Similarly, CPD co-ordinators could ensure that they use collaboration (e.g. in refining learning goals) as an important tool for teachers facing mandatory programmes - to develop ownership and personalise their learning.

Most of the effective CPD in the research included learning which took place in the teachers' own schools and classrooms. CPD leaders and teachers should consider harnessing all available in-school opportunities for professional learning: for example, through team teaching and through ensuring that lesson planning takes place collaboratively and is structured to include opportunities for debriefing.

The positive outcomes reported in the studies in the review were linked to CPD interventions which combined reflection with active experimentation. CPD leaders and head teachers may wish to review CPD plans and opportunities to ensure that opportunities for professional dialogue are linked to opportunities to experiment with new approaches in order to root learning conversations in classroom evidence. Teachers should consider seeking out such opportunities.

Collaborative CPD can be effective in more intimate settings. School and CPD leaders and CPD providers might want to consider offering teachers opportunities for small-group or paired work within any larger groupings.

There was little evidence about teachers' perceptions of the impact of the CPD on their pupils' learning in the studies which focused only on impact on the teachers. But studies from previous reviews by this Review Group that do contain pupil impact data highlighted the way in which pupil impact motivates teachers to sustain their learning. CPD leaders and programme managers should consider encouraging teachers to articulate, record and reflect upon their perceptions about the impact of the CPD and related changes in classroom practice on their students' learning.

The studies which focused solely on collecting teacher impact data were sustained over much longer periods than those which also collected student data. However, an earlier review found that gains for the CPD were not necessarily greater for those lasting more than one term. CPD leaders and head teachers may wish to reflect regularly on the match between the distance to be travelled and the length of any CPD interventions, while bearing in mind the benefits associated with CPD that lasts at least one term.

The CPD processes linked with positive outcomes for teachers in the studies with teacher-only data are consistent with those that show positive impact for pupils. This may suggest that these characteristics of CPD, in combination, could be used by school and CPD leaders, on an experimental basis, as proxy success indicators in weighing up whether to pursue certain CPD opportunities. Policy-makers could encourage schools and CPD providers to consider the highlighted characteristics of CPD as a set of questions to be applied to CPD proposals and activities in order to probe the likelihood of positive outcomes for students and teachers. Such approaches will be experimental and their usability and utility should be monitored.

The review found that studies which focused their aims on both teacher and student outcomes were more likely to have rooted their interventions in evidence about pedagogy. Conversely, studies which focused their aims on teacher impact were more likely to have been rooted in the literature about CPD and adult learning. CPD providers and CPD school leaders may wish to ensure that CPD programmes draw explicitly on both the relevant public knowledge bases about teaching and learning and about CPD.

Implications for researchers

The aims of the studies in the groups differed markedly. In the group of studies which collected data on both pupils and teachers (N=26), only three specifically targeted affective outcomes from their interventions. In the other group of studies, which collected data only on teacher impact, most (7 out of 11) targeted such outcomes. However, affective outcomes featured as incidental findings in many of the first group of studies. Researchers exploring the impact of CPD on teaching and learning should consider collecting systematic evidence about the impact of CPD on affective aspects of teachers' professional identity.

Studies of CPD which was linked to positive outcomes identified core elements of collaboration which recurred in combination. The specific effects of the individual components in isolation from each other were not explored. Researchers exploring the impact of CPD should consider collecting data about the relative impact of each of these core elements, by treating the components as independent variables.

Studies which focused on teacher data were less comparative in their designs than studies which collected both teacher and student data. While recognising that the control and comparison groups in the first group of studies comprised students rather than teachers, we nevertheless believe that studies which focus on teachers need to place greater emphasis on collecting comparative data.

The studies which focused solely on collecting teacher impact data were sustained over much longer periods than those which also collected student data. Future reviews could explore whether this difference is accounted for by the much greater emphasis on affective goals, or by the need for a short-term focus in order to enable collection of data about outcomes for particular cohorts of students.

The review found that the CPD featured in studies which focused their aims on both teacher and student outcomes was more likely to be rooted in evidence about pedagogy. Conversely, the CPD in studies which collected teacher-only impact data was more likely to be rooted in the literature about CPD and adult learning. Studies of CPD and the related interventions should consider incorporating and building equally upon the pedagogic and the CPD literature. In other words, studies that evaluate specific CPD programmes should problematise the nature of the changes in pedagogic practice as well as the CPD processes. Similarly, studies of the development of teaching and learning need to problematise the CPD processes and interventions, and to collect and analyse data about them if they are to provide research users with the information they need to operationalise findings and recommendations.

Implications for policy-makers

CPD is the vehicle through which all new policies must work if change is to become embedded rather than cosmetic. The cumulative picture of positive outcomes for teachers and pupils emerging from this series of reviews suggests that collaborative CPD between teachers has the potential to play a critical role in interpreting and embedding all policy initiatives in practice. The complex combinations of sustained peer and specialist support, of in-class experimentation coupled with protected space for reflection and structured dialogue, and the role of collaboration in personalising goals, sustaining commitment and developing ownership, are all challenging. They sit at some distance from traditional conceptions of CPD and the current arrangements for organising and evaluating it in many schools. However, they reinforce the emerging consensus about the nature of a proactive, modern profession within which teachers are seen as an important resource for each other in supporting and sustaining the development of their own and their colleagues' practice. Policy-makers should consider reviewing both explicit and implicit assumptions about the ways in which new initiatives are implemented in schools and how these may be enhanced by an explicit commitment to sustained, collaborative CPD.

References

Day C (1999) Developing Teachers: The challenges of lifelong learning. London: Falmer Press.

This report should be cited as: Cordingley P, Bell M, Evans D, Firth A (2005) The impact of collaborative CPD on classroom teaching and learning. Review: What do teacher impact data tell us about collaborative CPD? In: Research Evidence in Education Library. London: EPPI Centre, Social Science Research Unit, Institute of Education, University of London.