There are three tools that were used in these workshops:
12 questions to help you make sense of a process evaluation
10 questions to help you make sense of an outcome evaluation
10 questions to help you make sense of a review
12 Questions to help you make sense of a process evaluation
General comments
Three
broad issues need to be considered when appraising a process evaluation. Does the study tell you how the intervention was set up and monitored? Does it tell you what resources are necessary for an intervention? Does it tell you whether the intervention was acceptable to everyone involved?
· The
12 questions on the following pages are designed to help you think about these
issues systematically. The first three questions are screening questions to
identify poor quality studies and can be answered quickly. If the answer to
all three is "yes", it is worth proceeding with the remaining questions.
· There
is a fair degree of overlap between several of the questions.
· You
are asked to record a "yes", "no" or "can't tell" to most of the questions
· A
number of italicised prompts are given after each question. These are designed
to remind you why the question is important. There will not be time in the small
groups to answer them all in detail!
A/ Does the study focus on the delivery of a health
promotion intervention? |
Screening Questions |
|
1 Does the study focus on a health promotion intervention? |
Yes
Can't tell
No
|
A health promotion intervention aims
to reduce the risk if ill health, enable early treatment by screening, minimise
ill health or prevent the recurrence of ill health through:
- health education
- disease prevention
- health protection
|
|
2 Does the intervention have clearly stated aims? |
Yes
Can't tell
No
|
Aims are clearly focused if they describe:
- the target population
- the intervention
- the expected improvement in health
status
|
|
3 Does the study describe the key processes involved in delivering
this intervention? |
Yes
Can't tell
No
|
The processes involved may include:
- planning and consultation
- developing materials
- education and training
- establishing access to the target
population
- media and publicity
|
|
Detailed Questions |
|
4 Does the study tell you enough about planning and consultation? |
Yes
Can't tell
No
|
Planning and consultation could include:
- checking the need for health promotion
- seeking the views and knowledge of
the target group
- checking what resources are needed
and available to deliver the intervention
|
|
5 Does the study tell you enough about the collaborative effort
required for the intervention?
Are we told which individuals and/or
groups were working together to deliver an intervention (such as multidisciplinary
teams) or to enable people to take responsibility for their own health
(such as in community developments)?
|
Yes
Can't tell
No
|
6 Does the study tell you enough about the materials used in the
intervention? |
Yes
Can't tell
No
|
Considerable effort may be made to develop
audio, visual and printed material. Does the study describe these and report
how they were developed and disseminated?
|
|
7 Does the study tell you enough about how the target population
was identified and recruited? |
Yes
Can't tell
No
|
Reaching the target population with
the intervention may not be easy. Details of this and how they were introduced
to the study and invited to consent to the study should also be included |
|
8 Does the study tell you enough about education and training? |
Yes
Can't tell
No
|
Is the experience, education and training
described for all those involved in the study?
- those leading the intervention?
- all those delivering the intervention?
- those receiving the intervention? |
|
B/ What are the results?
|
|
8 Were all the processes described and adequately monitored? |
Yes
Can't tell
No
|
Does the study tell you:
- how successful they were in recruiting
people to deliver the intervention?
- how successful they were in training
people to deliver the intervention?
- how successful they were in reaching
the target population
|
|
9 Was the intervention acceptable? |
Yes
Can't tell
No
|
Some people may not have received the
intervention or responded to the intervention because they didn't like it.
Was it acceptable to:
- those delivering the intervention?
- those receiving the intervention?
|
|
C/ Will the results help me? |
10 Can the results be applied to the local population? |
Yes
Can't tell
No
|
Do you think that the people receiving
this intervention are similar enough to your population?
|
|
11 Were all important processes considered? |
Yes
Can't tell
No
|
If not, does this affect the decision? |
|
12 If you wanted to know whether this intervention promotes health
what outcomes would you want to measure? |
|
After all the effort of planning a new
intervention, overcoming difficulties in the delivery and asking people
whether they like it, there is still the question of whether it actually
works.
|
|
10 questions to help you make sense of an outcome evaluation
General comments
· Three broad issues need to be considered
when appraising an outcome evaluation. Are the results of the study valid?
What are the results? Will the results help me?
· The 10 questions on the following
pages are designed to help you think about these issues systematically. The
first three questions are screening questions and can be answered quickly. If
the answer to all three is "yes", it is worth proceeding with the remaining
questions.
· There is a fair degree of overlap
between several questions.
· You are asked to record a "yes",
"no" or "can't tell" to most of the questions.
· A number of italicised prompts are
given after each question. These are designed to remind you why the question
is important or what you should look for. There will not be time in the small
group to answer them all in detail! Individual studies do not necessarily address
all the issues - you need to decide whether omitting to address and issue undermines
the validity of the study or only narrows its scope.
· The 10 questions are adapted from:
Guyatt GH, Sackett DL, Cook DJ, Users' guides to the medical literature. II.
How to use an article about therapy or prevention. JAMA 1993; 270: 2598-2601.
A/ Are the results of the outcome evaluation valid? |
Screening Questions |
|
1 Did the evaluation address a clearly focused issue? |
Yes
Can't tell
No
|
An issue can be 'focused' in terms of
- the population studied
- the intervention given
- the outcomes considered
|
|
2 Were the people receiving the intervention compared with an
equivalent control or comparison group? |
Yes
Can't tell
No
|
A control or comparison group
may be equivalent if
- the people in the different groups
were selected in similar ways,
- descriptions of the different groups
of people (demographic data) were very similar, or
- the people were allocated to the
different groups randomly
|
|
3 Were all of the people who entered the evaluation properly accounted
for and attributed at its conclusion? |
Yes
Can't tell
No
|
Was follow up complete? Look for
- the number of people recruited (participation
rate)
- the number of people allocated to
the different groups
- the number of people reported in
the outcome data tables
- the number of people who dropped
out (attrition rate) and what we are told about them
|
|
Detailed Questions |
|
4 Was the intervention described clearly? |
Yes
Can't tell
No
|
Would it be possible to replicate the
intervention from this description?
Aside from the experimental intervention,
were the groups treated equally?
|
|
5 Is it clear how the control group and experimental groups did
or did not change after the intervention?
Is data given on the outcome measures
for all groups of people both before and after the intervention?
|
Yes
Can't tell
No
|
B/What are the results?
|
|
6 How large was the impact of the intervention? |
|
What outcomes are measured?
How large was the difference, if any,
for each of the outcomes measured?
|
|
7 How precise are the results? |
|
What are the confidence limits for each
result reported? |
|
C/ Will the results help me? |
8 Can the results be applied to the local population? |
Yes
Can't tell
No
|
Do you think that the people involved
in the evaluation are similar enough to your population? |
|
9 Were all important outcomes considered? |
Yes
Can't tell
No
|
If not, does this affect your decision? |
|
10 Are the benefits worth the harms and costs? |
Yes
Can't tell
No
|
This is unlikely to be addressed by
the evaluation. But what do you think? |
|
10 questions to help you make sense of a review
General comments
· Three broad issues need to be considered
when appraising a review article.
Are the results of the review valid?
What are the results?
Will the results help locally?
· The 10 questions on the following
pages are designed to help you think about these issues systematically. The
first two questions are screening questions and can be answered quickly. If
the answer to both is "yes", it is worth proceeding with the remaining questions.
· There is a fair degree of overlap
between several of the questions.
· You are asked to record a "yes",
"no" or "can't tell" to most of the questions.
· A number of italicised prompts are
given after each question. These are designed to remind you why the question
is important. There will not be time in the small groups to answer them all
in detail!
· The 10 questions are adapted from:
Oxman AD, Guyatt GH et al, Users' Guides to The Medical Literature, VI How to
use an overview. (JAMA 1994; 272 (17): 1367-1371)
A/ Are the results of the review valid? |
Screening Questions |
|
1. Did the review address a clearly focused issue? |
Yes
Can't tell
No
|
A issue can be 'focused' in terms of
- the population studied
- the intervention given
- the outcomes considered
|
|
2 Did the authors select the right sort of studies for the review? |
Yes
Can't tell
No
|
The 'right sort of studies' would
- address the review's question
- have an adequate study design
|
|
Detailed Questions |
|
3 Do you think the important, relevant studies were included? |
Yes
Can't tell
No
|
Look for
- which bibliographic databases were
used
- checks from reference lists
- personal contact with experts
- search for unpublished as well as
published studies
- search for non-English language studies
|
. |
4 Did the review's authors do enough to assess the quality of
the included studies? |
Yes
Can't tell
No
|
The authors need to consider the rigour
of the studies they have identified. Lack of rigour may affect the studies'
results (All that glistens is not gold!)
|
|
5 Were the results similar from study to study? |
Yes
Can't tell
No
|
Consider whether
- the results of all the included studies
are clearly displayed
- the results of the different studies
are similar
- the reasons for any variations in
results are discussed
|
|
B/ What are the results? |
|
6 What is the overall result of the review? |
|
Consider
- if you are clear about the review's
'bottom line' results
- what these are (numerically if appropriate)
- what units these results are expressed
in
|
|
7 How precise are the results?
Are there confidence limits? What are
they?
|
|
C/ Will the results help locally? |
8 Can the results be applied to the local population? |
Yes
Can't tell
No
|
Do you think that the people covered
by the review are similar enough to your population?
|
|
9 Were all important outcomes considered? |
Yes
Can't tell
No
|
If not, does this affect the decision?
|
|
10 Are the benefits worth the harms and costs? |
Yes
Can't tell
No
|
This is unlikely to be addressed by
the review. But what do you think?
|
|