This page describes the learning from EPPI Centre involvement in workshops on systematic review techniques

PHASE [1]

A report on a series of workshops on evidence-based information made the following learning points:

• Whilst involving participants with a broad range of skills in CASP workshops has encouraged peer supported learning, this did not happen with workshops attended by health promotion specialist from different organisations.
• Workshop participants were more eager to discuss the relevance of interventions being studied, or the political background/ ethos of the intervention rather than the quality of evaluation.
• There is a dearth of material which stands up well to critical appraisal let alone matches principles encapsulated in the Ottowa Charter for health promotion5.
• Workshop participants would find it helpful to be able to directly compare studies of different qualities.
• Participants value particularly highly well-designed materials for use in workshops and afterwards.
• There is a need for background papers discussing evidence-based health promotion
• Participants considered the Cochrane Database demonstration interesting but irrelevant to their work.
• Participants responded very positively to the opportunity to write to the review author with constructive criticism.
• Some participants respond positively to the concepts of evidence-based health promotion when asked to explore them in terms of their own experience.
• The difficulties encountered in conducting controlled trials in some settings are interpreted by some as impossibilities.
• The purpose of randomisation in producing similar groups of diverse people or settings, and thereby accounting for their individuality rather than dismissing it, is poorly understood by some workshop participants and should be addressed in a separate workshop or literature pack.
• Apparent support for evidence-based health promotion may be weakened by discussing evidence of effectiveness in focused terms. This may be related to poor understanding of the purpose of randomisation in producing similar groups of diverse people or settings.
• Those sources of information noted for reliable evidence of effectiveness are used least and are considered irrelevant because of poor coverage of health promotion, or inappropriate because of their high reliance on controlled trials and, in some cases, narrow outcome measures.
• Those sources of information which are noted for their relevance of health promotion, are poor in coverage of effectiveness information.
• The competitive environment of the NHS internal market discourages both discussion of evidence based health promotion and attempts to develop it within contracting cycles.
• Misconceptions about the skills required to establish effective health promotion programmes may discourage practitioners from learning how to integrate research findings into their work.
• There is considerable enthusiasm amongst some purchasers and providers for working in partnership with researchers to advance evidence-based health promotion
• Differences in interpretation of terms related to evidence of effectiveness suggest that additional methods for evaluating the effect of workshops on participants’ attitudes should be considered.

Recommendations included:

• Expanding the evidence base of health promotion
• Encouraging reference to sources of evidence of effectiveness
• Fostering evidence-based health promotion
• Developing PHASE workshops.

The HIVSA project [2]

The HIVSA project was a series of workshops in southern Africa with three aims:

1. to develop and deliver participatory workshops to support evidence-informed decision-making by policy-makers, practitioners and researchers involved in designing, implementing and/or evaluating educational programmes for HIV prevention in southern Africa;
2. to develop with participants at the workshops a web-based register of published and unpublished evidence, drawn from studies of educational interventions planned, conducted and/or evaluated in southern Africa;
3. to use the web-based register of evidence to conduct a systematic review of educational interventions for HIV prevention in southern Africa.

The project  has raised some important questions about the processes involved in developing critical appraisal skills:

• How realistic is it to expect people to participate in comprehensive systematic reviews when they are already committed to a full-time job that may not support their training or research? Indeed, systematic reviews normally take one year to complete and another year to disseminate.
• What is the value of less rigorous yet policy- and practice-informing syntheses, such as the ‘mini-reviews’ conducted as ‘decision-making syntheses’ by HIVSA participants? Perhaps this is a question that highlights the lack of congruity between what researchers might call 'rigorous' and what practitioners would call 'useful'.
• How best can the value of ‘indirect research use’ in policy-making and practice be captured? In particular, it has not been empirically demonstrated, although it might well be argued, that supporting participants to conduct their own ‘decision-making syntheses’ may have informed and influenced their practice by encouraging critical thinking. Critical thinking is an integral part of training in systematic review methodologies, because even though conducting a review is in itself an exercise in review research, the growth reviewers experience throughout the process requires closer observation if we are to better understand how this might occur.

Recommendations from the workshops included:

• The HIVSA workshops provided a very successful model for training mixed groups of participants. When running workshops for participants from different backgrounds and in different fields, there is a need to take into account the different needs and expectations of all those participating. This is particularly true when working with policy-makers and practitioners whose needs and skills may be more varied than those of researchers.
• Email and the internet were utilised for the recruitment of participants for the HIVSA workshops very successfully. It is important to be prepared for unreliable internet connections, and ensure you have alternative means of communicating (such as fax and phone).
• Particular effort should go into making clear the aims, content and outcomes of the workshops, to all those wishing to attend in advance of the workshops.
• When teaching the same material to consecutive groups of participants, each group must be considered separately, and any differences incorporated into the design of the workshops. Sufficient time must be allowed before each workshop for this preparation by the facilitators.
• The demands of other work on the participants should be taken into account, at the planning stage, and where possible arrangements made for email and fax facilities to be made available. The need to take such practicalities into account is particularly likely to occur when working with policy-makers or senior members of organisations.
• It is important to be transparent in advance of the workshops about what funding is available for participants, spelling out very specifically, what is covered by the organisers, and what participants are expected to pay for themselves.
• Detailed teaching materials, which can be referred to during and after the workshops, are popular with participants. Background reading should also be made available on as wide a range of literature as possible.
• The participatory nature of the HIVSA workshops could be considered one of their major strengths. Such participatory training methods are considered appropriate for teaching adults in an approach such as evidence-based decision-making. Individuals should be encouraged to share their experiences and skills with the group, through paired and small group tasks, as well as with the whole group. The facilitators' role must be to introduce topics and guide discussions.
• The use of locally relevant examples in the HIVSA workshops helped to gain participants' interest in, and understanding of, the workshop content. Participants might be invited to think of examples of relevant work in their area in advance of a session, and then encouraged to share and draw upon these in completing a task or taking part in a discussion.

1. PHASE: Promoting Health After Sifting the Evidence  (1966)

2. The HIVSA training manual (2001)