PublicationsSystematic reviewsPHASEPHASE: tools
Promoting health after sifting the evidence: tools
There are three tools that were used in these workshops:

12 questions to help you make sense of a process evaluation

10 questions to help you make sense of an outcome evaluation

10 questions to help you make sense of a review



12 Questions to help you make sense of a process evaluation

General comments

Three broad issues need to be considered when appraising a process evaluation. Does the study tell you how the intervention was set up and monitored? Does it tell you what resources are necessary for an intervention? Does it tell you whether the intervention was acceptable to everyone involved?

· The 12 questions on the following pages are designed to help you think about these issues systematically. The first three questions are screening questions to identify poor quality studies and can be answered quickly. If the answer to all three is "yes", it is worth proceeding with the remaining questions.

· There is a fair degree of overlap between several of the questions.

· You are asked to record a "yes", "no" or "can't tell" to most of the questions

· A number of italicised prompts are given after each question. These are designed to remind you why the question is important. There will not be time in the small groups to answer them all in detail!


A/ Does the study focus on the delivery of a health promotion intervention?
Screening Questions  
1 Does the study focus on a health promotion intervention?

Yes

Can't tell

No

A health promotion intervention aims to reduce the risk if ill health, enable early treatment by screening, minimise ill health or prevent the recurrence of ill health through:

- health education

- disease prevention

- health protection

 
2 Does the intervention have clearly stated aims?

Yes

Can't tell

No

Aims are clearly focused if they describe:

- the target population

- the intervention

- the expected improvement in health status

 
3 Does the study describe the key processes involved in delivering this intervention?

Yes

Can't tell

No

The processes involved may include:

- planning and consultation

- developing materials

- education and training

- establishing access to the target population

- media and publicity

 


Detailed Questions  
4 Does the study tell you enough about planning and consultation?

Yes

Can't tell

No

Planning and consultation could include:

- checking the need for health promotion

- seeking the views and knowledge of the target group

- checking what resources are needed and available to deliver the intervention

 
5 Does the study tell you enough about the collaborative effort required for the intervention?

Are we told which individuals and/or groups were working together to deliver an intervention (such as multidisciplinary teams) or to enable people to take responsibility for their own health (such as in community developments)?

Yes

Can't tell

No

6 Does the study tell you enough about the materials used in the intervention?

Yes

Can't tell

No

Considerable effort may be made to develop audio, visual and printed material. Does the study describe these and report how they were developed and disseminated?  
7 Does the study tell you enough about how the target population was identified and recruited?

Yes

Can't tell

No

Reaching the target population with the intervention may not be easy. Details of this and how they were introduced to the study and invited to consent to the study should also be included  
8 Does the study tell you enough about education and training?

Yes

Can't tell

No

Is the experience, education and training described for all those involved in the study?
- those leading the intervention?
- all those delivering the intervention?
- those receiving the intervention?
 

 

B/ What are the results?

 
8 Were all the processes described and adequately monitored?

Yes

Can't tell

No

Does the study tell you:

- how successful they were in recruiting people to deliver the intervention?

- how successful they were in training people to deliver the intervention?

- how successful they were in reaching the target population

 
9 Was the intervention acceptable?

Yes

Can't tell

No

Some people may not have received the intervention or responded to the intervention because they didn't like it. Was it acceptable to:

- those delivering the intervention?

- those receiving the intervention?

 

 

C/ Will the results help me?
10 Can the results be applied to the local population?

Yes

Can't tell

No

Do you think that the people receiving this intervention are similar enough to your population?  
11 Were all important processes considered?

Yes

Can't tell

No

If not, does this affect the decision?  
12 If you wanted to know whether this intervention promotes health what outcomes would you want to measure?  
After all the effort of planning a new intervention, overcoming difficulties in the delivery and asking people whether they like it, there is still the question of whether it actually works.  


10 questions to help you make sense of an outcome evaluation

General comments

· Three broad issues need to be considered when appraising an outcome evaluation. Are the results of the study valid? What are the results? Will the results help me?

· The 10 questions on the following pages are designed to help you think about these issues systematically. The first three questions are screening questions and can be answered quickly. If the answer to all three is "yes", it is worth proceeding with the remaining questions.

· There is a fair degree of overlap between several questions.

· You are asked to record a "yes", "no" or "can't tell" to most of the questions.

· A number of italicised prompts are given after each question. These are designed to remind you why the question is important or what you should look for. There will not be time in the small group to answer them all in detail! Individual studies do not necessarily address all the issues - you need to decide whether omitting to address and issue undermines the validity of the study or only narrows its scope.

· The 10 questions are adapted from: Guyatt GH, Sackett DL, Cook DJ, Users' guides to the medical literature. II. How to use an article about therapy or prevention. JAMA 1993; 270: 2598-2601.


A/ Are the results of the outcome evaluation valid?
Screening Questions
1 Did the evaluation address a clearly focused issue?

Yes

Can't tell

No

An issue can be 'focused' in terms of

- the population studied

- the intervention given

- the outcomes considered

 
2 Were the people receiving the intervention compared with an equivalent control or comparison group?

Yes

Can't tell

No

A control or comparison group may be equivalent if

- the people in the different groups were selected in similar ways,

- descriptions of the different groups of people (demographic data) were very similar, or

- the people were allocated to the different groups randomly

 
3 Were all of the people who entered the evaluation properly accounted for and attributed at its conclusion?

Yes

Can't tell

No

Was follow up complete? Look for

- the number of people recruited (participation rate)

- the number of people allocated to the different groups

- the number of people reported in the outcome data tables

- the number of people who dropped out (attrition rate) and what we are told about them

 



Detailed Questions  
4 Was the intervention described clearly?

Yes

Can't tell

No

Would it be possible to replicate the intervention from this description?

Aside from the experimental intervention, were the groups treated equally?

 
5 Is it clear how the control group and experimental groups did or did not change after the intervention?

Is data given on the outcome measures for all groups of people both before and after the intervention?

Yes

Can't tell

No



B/What are the results?


6 How large was the impact of the intervention?
What outcomes are measured?

How large was the difference, if any, for each of the outcomes measured?





7 How precise are the results?
What are the confidence limits for each result reported?






C/ Will the results help me?
8 Can the results be applied to the local population?

Yes

Can't tell

No

Do you think that the people involved in the evaluation are similar enough to your population?







9 Were all important outcomes considered?

Yes

Can't tell

No

If not, does this affect your decision?







10 Are the benefits worth the harms and costs?

Yes

Can't tell

No

This is unlikely to be addressed by the evaluation. But what do you think?








10 questions to help you make sense of a review

General comments

· Three broad issues need to be considered when appraising a review article.

Are the results of the review valid?

What are the results?

Will the results help locally?

· The 10 questions on the following pages are designed to help you think about these issues systematically. The first two questions are screening questions and can be answered quickly. If the answer to both is "yes", it is worth proceeding with the remaining questions.

· There is a fair degree of overlap between several of the questions.

· You are asked to record a "yes", "no" or "can't tell" to most of the questions.

· A number of italicised prompts are given after each question. These are designed to remind you why the question is important. There will not be time in the small groups to answer them all in detail!

· The 10 questions are adapted from: Oxman AD, Guyatt GH et al, Users' Guides to The Medical Literature, VI How to use an overview. (JAMA 1994; 272 (17): 1367-1371)

A/ Are the results of the review valid?
Screening Questions  
1. Did the review address a clearly focused issue?

Yes

Can't tell

No

A issue can be 'focused' in terms of

- the population studied

- the intervention given

- the outcomes considered

 
2 Did the authors select the right sort of studies for the review?

Yes

Can't tell

No

The 'right sort of studies' would

- address the review's question

- have an adequate study design

 


Detailed Questions  
3 Do you think the important, relevant studies were included?

Yes

Can't tell

No

Look for

- which bibliographic databases were used

- checks from reference lists

- personal contact with experts

- search for unpublished as well as published studies

- search for non-English language studies

.
4 Did the review's authors do enough to assess the quality of the included studies?

Yes

Can't tell

No

The authors need to consider the rigour of the studies they have identified. Lack of rigour may affect the studies' results (All that glistens is not gold!)  
5 Were the results similar from study to study?

Yes

Can't tell

No

Consider whether

- the results of all the included studies are clearly displayed

- the results of the different studies are similar

- the reasons for any variations in results are discussed

 
B/ What are the results?  
6 What is the overall result of the review?  
Consider

- if you are clear about the review's 'bottom line' results

- what these are (numerically if appropriate)

- what units these results are expressed in



 
7 How precise are the results?

Are there confidence limits? What are they?





 
C/ Will the results help locally?
8 Can the results be applied to the local population?

Yes

Can't tell

No

Do you think that the people covered by the review are similar enough to your population?





 
9 Were all important outcomes considered?

Yes

Can't tell

No

If not, does this affect the decision?





 
10 Are the benefits worth the harms and costs?

Yes

Can't tell

No

This is unlikely to be addressed by the review. But what do you think?





 

  
Copyright 2019 Social Science Research Unit, UCL Institute of Education :: Privacy Statement :: Terms Of Use :: Site Map :: Login
About::Projects::Publications::Research Use::Courses & seminars::Resources::Databases::Blog