Many humanitarians are evidence-aware, but may find it difficult to draw on what is known or find knowledge that speaks to their context. They may also be pressed for time to find or judge the relevance of what is often a dispersed literature. To address this gap the Humanitarian Evidence Programme, a partnership between Oxfam and Feinstein International Center at Tufts University, published eight systematic reviews in areas identified as a priority by humanitarian policy and practitioner stakeholders. Typical of the sector, and similar to international development, decision-makers ask very broad questions. Kelly Dickson and Mukdarut Bangpan reflect on the challenges we encountered when producing a mixed methods evidence synthesis for this programme, on mental health and psychosocial programmes for people affected by humanitarian emergencies.
Broad policy questions need comprehensive and flexible conceptual frameworks.
The humanitarian sector is a diverse field of inquiry. It can include earthquakes and typhoons, protracted conflict, acts of terrorism, biological hazards, and other natural and man-made disasters. Our brief, to consider the effectiveness and implementation of programmes addressing the mental health and psychosocial well-being of both children and adults, was similarly broad. To guide the review, we needed a comprehensive and flexible conceptual framework which was extensive enough to capture the diversity of definitions that can be found, but also specific enough to guide our reviewing task; from searching, screening, to shaping the synthesis. To achieve this, we drew heavily on existing research literature to provide us with widely recognised definitions of humanitarian emergencies and outcomes (e.g. post-traumatic stress syndrome, depression, anxiety, social support). We were also able to draw on existing guidelines on Mental Health and Psychosocial Support in Emergency Settings, which had benefitted from stakeholder input from United Nations agencies, NGOs and academic institutions. These guidelines provided an intervention framework, in the form of a 'layered system of MHPSS support', which was used to map and signpost studies in the review and supported useful identification of gaps in the evidence-base.
Emerging research areas need tailored search strategies…
Like for most systematic reviews we aimed to conduct a comprehensive search of the literature to identify the best available evidence to answer the review questions. We found that, similar to previous reviews, relying on bibliographic databases is not sufficient. Although some areas of research, such as the effects of clinical treatment, are well indexed in databases, other areas, such as studies of delivery and receipt of mental health programmes, were more often found by hand-searching, a time consuming but productive activity.
…and inclusive but transparent approaches to judging rigour and relevance.
One of the defining characteristics of systematic reviews is not only the transparent and accountable methods used to identify relevant literature, but the judgements that are made about the quality, and thus trustworthiness of the evidence to inform policy and practitioner decision-making. These judgements can be based on standardised tools for judging the methodological quality of evidence or those designed specifically for this review. A common criticism of systematic reviews, and a concern for the humanitarian evidence programme, was the potential to miss important lessons from studies of lower quality. Our approach was to two-fold. The first was to retain all studies, but to ‘grade’ the evidence informing the question on impact taking into consideration study quality rating, number of studies, and effect sizes, before making summary statements on the direction of effect (e.g. there is strong, moderate or limited evidence). The second was to judge the relevance of studies with qualitative evidence informing our question on implementation, clearly signposting this through the evidence synthesis. This approach, draws on our institutional learning, and addressed concerns from policy and practitioners keen to use evidence.
Systematic reviews shaped by humanitarian organisations are best shared through humanitarian organisations.
Not only did we draw on questions and concepts defined by humanitarian organisations, but we have also found them essential for sharing the findings. Our interpersonal relationships and networks (see Langer et al., 2016) have increased the visibility of the review findings. Through working closely with others as part of a ‘community of practice’ we were able to discuss the current evidence base and implications for the future, and found websites, conferences and webinars to suit wider policy and practice networks.
Producing mixed-methods evidence synthesis in a new arena such as the humanitarian evidence sector has presented us with an exciting opportunity to apply our skills and adapt synthesis methodology to benefit the field. This has required immersing ourselves in the policy literature to develop an appropriate conceptual framework, drawing on our technical expertise to identify and appraise studies and building new social networks to support dissemination activities. Doing so has, once again, enabled us draw on our insider knowledge of working at the interface of policy and practice to relevant and accessible evidence. The result is evidence where both relevance and rigour are maximised to suit decisions made in challenging circumstances.
About the authors
Mukdarut Bangpan is a research officer at the EPPI-Centre with many years' experience of conducting systematic review and supporting review teams, particularly in the area of international development.
Kelly Dickson is Research at the EPPI-Centre, experienced in conducting mixed methods systematic reviews and building research synthesis capacity across disciplines, including international development. She is also a psychotherapist with an interest in mental health and psychosocial wellbeing of adults and children.
Bangpan M, Lambert F, Chiumento A, Dickson K (2016). The impact of mental health and psychosocial support programmes for populations affected by humanitarian emergencies: a systematic review protocol. Oxford: Oxfam.
Interagency standing committee (2007). IASC guidelines on mental health and psychosocial support in emergency settings. Inter-Agency Standing Committee. https://interagencystandingcommittee.org/
Langer L, Tripney J, Gough D (2016). The science of using science: researching the use of Research evidence in decision-making. London: EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London