About this toolkit

Who is this toolkit for? This toolkit is for policy-makers who want to engage with evidence and with stakeholders in their decision-making process. And for impact evaluators or evidence synthesis teams who want to engage stakeholders in the production and use of their research. Their relevant evidence may come from formal studies or from stakeholders – policy makers, those people required to implement decisions taken, those people affected by the decisions and researchers.

What is this toolkit for? Policy decisions – taken by international agencies, national governments or local organisations operating in complex socio-political contexts – can sometimes be informed by evidence of how well policies or programmes work (the impact they have, and how much this varies); evidence about how to implement them; or evidence about the context in which they are to be applied. Conversely, those complex socio-political contexts mean these decisions are routinely taken against a backdrop of uncertainty. That may be the uncertainty of how well policies or programmes work (the impact they have, and how much this varies); uncertainty about how to implement them; or uncertainty about the context in which they are to be applied, particularly when it is unfamiliar or changing. This toolkit signposts tools and research about finding and making sense of evidence and recognising the remaining uncertainty when making policy decisions.

Making judgements about whether evidence from formal studies might be relevant to local, current problems, and what might be acceptable or feasible in the face of remaining uncertainty, requires context-specific know-how and tacit knowledge (henceforth referred to as locally rooted evidence). This toolkit signposts tools and research about finding different groups of stakeholders and engaging them with problems, evidence and each other when making policy decisions.

Facing uncertainty is a challenge when evaluating impact or synthesising evidence too. Impact evaluations generate evidence (thereby reducing uncertainty) about the scale of effects. When they are combined in systematic reviews, uncertainty about how much impact varies can be assessed with a reassuring level of rigour, thereby providing generalisable evidence. Doing this work well involves tackling other uncertainties: which interventions are most important to evaluate, how is success or failure recognised, and what are implications do the findings have for the wider world? This toolkit signposts tools and research about finding stakeholders and engaging them with the research process and each other when planning or conducting research, or interpreting the findings to make actionable recommendations.

Availability and reliability of both generalisable and locally rooted evidence varies enormously across different settings and in relation to different decisions. Effective stakeholder engagement can enhance quality and access to one or other (or both!) and ideally bring them into balance for more effective decision-making.

How might stakeholder engagement help? By its nature, locally rooted evidence is often tacit and held within the minds of individual beneficiaries, practitioners and policy makers. Here, stakeholder engagement activities can help tease out this context-specific knowledge – making it explicit, appraisable in relation to specific decisions, and actionable. Stakeholder engagement activities focused on generalisable evidence allow stakeholders to assess the relevance of that evidence to the task at hand. Once relevant knowledge has been brought together from both formal studies and relevant stakeholders, that knowledge can be applied to making policy decisions or to conducting research.

Why is this toolkit needed? A large number of methods for stakeholder engagement have evolved from different schools of practice, often adopting different terminology. Some methods such as Accountability to Affected Communities or Evidence-informed Guideline Development, are common in particular parts of the stakeholder engagement landscape. Other methods that share the same or similar names, such as action research or participatory action research, may be applied in distinctive ways. While other methods with distinct names can appear remarkably similar, for instance knowledge brokering and human centred design. Choosing between these methods can therefore be confusing. What’s more, it is not always clear which methods best suit different circumstances.

How does this toolkit help? This toolkit is designed to cut through this confusion and help toolkit users determine the most appropriate stakeholder engagement methods for the task at hand. It describes stakeholder engagement models that suit development interventions when:

1. Enhancing evidence use for policy decisions; or
2. Conducting or commissioning impact evaluations and research syntheses.

This toolkit signposts systematic review evidence and tools to help with:

• Finding and making sense of research evidence;
• Finding relevant stakeholders to draw on their knowledge;
• Helping stakeholders discuss issues and make decisions.

Where does it come from? As a team of professionals from both the evidence-for-decisions community and the development and humanitarian practitioner community, this toolkit comes from:

• Our own collective experience;
• Analysis of systematic reviews of stakeholder engagement; and
• Listening to people who work with other stakeholders to make decisions or conduct research.

The tools in this toolkit The toolkit comprises five different tools which can be used in different ways. They are all designed to help policy-makers and researchers choose appropriate methods and existing tools for engaging with evidence and each other depending on:

• Whether the task is to make a policy decision or plan some research
• Whether their decision or research is to apply only locally, or more generally
• How much clarity and consensus there is about what they know when starting out, whether this is being clear and agreed about where a decision will be applied, or clear and agreed about the existing knowledge that new research will build on.

In short, the choice of methods for making policy decisions depends, in part, on whether decision-makers and stakeholders have a shared understanding about the context where the decisions will be applied, and whether the research available is judged relevant to the circumstances.

Similarly, the choice of methods for making decisions about conducting research depends, in part, on whether the aim is to produce new knowledge that will be for local use or be applied much more widely, and whether the researchers and other stakeholders have a shared understanding of what is already known before the research begins.

Tool 1: A dynamic framework spanning the whole field of stakeholder engagement that explains the key dimensions that distinguish major differences in stakeholder engagement.
Tool 2: A flowchart illustrating pathways for choosing appropriate methods for stakeholder engagement.
Tool 3: An interactive guide that matches key contextual features of engagement methods to tasks and circumstances.
Tool 4: An evidence map that signposts systematic reviews and practical tools and guidance.
Tool 5: Guidance for identifying and understanding stakeholders and their relationships.

Tool 1: A framework illustrating the field of stakeholder engagement

Tool 1 is an analytical framework rather than a practical guide. It is designed for researching stakeholder engagement. It underpins the other four tools.

The double matrix design in figure 1 presents a vision of stakeholder engagement across the whole field. It is designed to understand the whole field of stakeholder engagement with evidence and uncertainty, its key dimensions and how the field varies.

This framework consists of two matrices, one focused on engaging stakeholders in making decisions, and the other focused on engaging stakeholders in conducting research; both set against a backdrop of varying social, cultural and political contexts.

The top row of each matrix represents situations where there is a shared understanding of the working context: the social context where policy decisions are to be implemented in the left hand matrix; and the knowledge context in which research is conducted in the right hand matrix. The bottom row of each matrix represents situations where the shared understanding of the working context is limited: where the social context of policy decisions are to implemented are unfamiliar, unpredictable or generally not well understood in the left hand matrix; and where the meaning of key concepts for research are unfamiliar, ambiguous or contested in the right hand matrix. The inner two columns, across the two matrices, represent working contexts where generalisable evidence is available for policy decisions (right hand matrix) or the aim when conducting research (right hand matrix).

This red triangle highlights how the social, cultural and political backdrop can vary in terms of how the world is understood, the relevant institutions and networks, and the power dynamics.

Tool 2: A flow chart to recognise the working context for a specific task

Tool 2 is a practical step-by-step guide for recognising a specific working context in order to choose appropriate stakeholder methods.

This flow chart (figure 2) navigates the framework with a series of choices about a specific task to be done, and about existing shared understanding and generalisable evidence.

Step 1: Is the task making a policy decision or conducting research?

Step 2, for policy decisions: Decide whether the context from implementing a decision is (a) clear and widely understood; or (b) unfamiliar, uncertain, ambiguous or contested?

Step 3, for policy decisions: Access generalisable evidence, appraise it with stakeholders and decide to discard it, devalue it or use it; working with a few key stakeholders following step 2(a) or many diverse stakeholders following step 2(b).

Step 4, for policy decisions: If generalisable evidence is adequate make a decision. If generalisable evidence is deficient request more research if time allows; and make a decision working with a few key stakeholders following step 2(a) or many diverse stakeholders following step 2(b).

Step 2, for conducting research: Decide whether the key concepts underpinning the research are (a) clear and widely understood; or (b) unfamiliar, uncertain, ambiguous or contested?

Step 3, for conducting research: Scope the research problem, particularly whether the aim is to produce generalisable evidence or locally tailored evidence.

Step 4, for conducting research: Conduct impact evaluation or synthesis: working with widespread stakeholders for generalisable evidence, or local stakeholders for locally tailored evidence; and working with a few key stakeholders following step 2(a) or many diverse stakeholders following step 2(b).

Following these steps, and this flow chart leads to understanding where any particular task sits within this landscape, and the appropriate general approach for stakeholder engagement. Once this is clear, the interactive guide (see Tool 3) can signpost users to the most appropriate engagement methods for the task concerned.

Tool 3: Interactive guide to match key features of engagement methods to tasks and circumstances

Tool 3 is an interactive guide to signpost the stakeholder methods of choice depending on the task and context of working.

This interactive guide comprises the two matrices that appear in framework in tool 1. In this interactive guide, clicking on any quadrant (numbered as A-D and 1-4) opens a pop-up with further information about suitable stakeholder engagement methods and links to relevant resources. Each pop-up describes:

• the task (making policy decisions or conducting research);
• the context of working (degree of clarity and consensus about knowledge)
• a generic description of methods of choice for this task and context
• key limitations
• a link to an example of relevant evidence
• a link to an example of a relevant tool

Tool 4: A map of evidence and tools for stakeholder engagement

This tool signposts evidence and tools to support stakeholder engagement methods that suit particular tasks and working contexts. To access the interactive map, follow this link:  Evidence map: engaging stakeholders with evidence and uncertainty. The map is best viewed in Google Chrome.

This tool is designed as a two-dimensional map. The left hand axis on the map matches the tasks (making decisions or conducting research) and circumstances of the stakeholder engagement framework (degree of clarity and consensus about knowledge). The top axis matches these stakeholder engagement activities:

• Developing stakeholders by
  • Identifying or recruiting relevant stakeholders
  • Improving their skills or motivation
• Engaging stakeholders with evidence
  • Facilitating their access to evidence
  • Fostering changes to decision-making structures and processes
• Engaging stakeholders with each other
  • Through consultation
  • Through discussion
  • Through shared decision-making

Evidence and tools can be found through three routes:

1. Choosing a row to match the task and working circumstances and a column to match engagement activities of interest. Where the row and column meet: click on the blue circles to find systematic review evidence about appropriate engagement methods; click on the green circle to find tools to support appropriate engagement methods.
2. Free text searching by clicking on ‘Reader’ above the map, and typing terms of interest in the ‘Filter’ box that appears above the list of references.
3. Combining these two approaches: follow step one with a free text search in the filter box that appears.


Figure 3: Screenshot of the map of evidence and tools for stakeholder engagement.

Tool 5: Guidance for identifying and understanding stakeholders and their relationships

Tool 5 is a selection of existing tools to help policy makers and researchers analyse the social and political context of their work.

What decisions are made and what research is conducted and used is influenced by political power, democratic processes, institutional mechanisms, values and priorities.

Methods of choice: identifying stakeholders who are essential for, or may influence, decision making or research processes.

Limitations: The political analysis skills required for understanding the roles and influence of stakeholders, and the technical skills required for producing generalisable knowledge, are usually held by different sets of people.

Tools: Every day political analysis and Collaboration mapping

Evidence: stakeholder mapping (Henwood 2017)

Acknowledgments

Please cite this toolkit as: Langer L, Nduku P, Umayam H, Conroy K, Maugham C, Bradley T, Bangpan M, Kneale D, Roche C, Oliver S (2020) Engaging stakeholders with evidence and uncertainty: a toolkit. London: EPPI Centre, UCL Social Research Institute, UCL Institute of Education, University College London.

This toolkit was developed by the Africa Centre for Evidence at the University of Johannesburg , and the EPPI-Centre at UCL Social Research Institute with UK Aid funding through the Centre for Excellence in Development Impact and Learning. The toolkit was transferred to the EPPICentre website with funding from The International Development Research Centre (IDRC - Canada).

^

The content of this toolkit is the sole responsibility of the authors and does not represent the opinions of CEDIL or the Foreign, Commonwealth & Development Office, UK, who funded the original work, or the International Development Research Centre, Ottawa, Canada who are supporting the maintenance of the toolkit.
Any errors and omissions are also the sole responsibility of the authors. Please direct any comments or queries to the corresponding author, Sandy Oliver.