Methods & ToolsMethods references
Methods references

Approaches to reviewing

User involvement
Different types of review
Methodological and other challenges
Text mining, automation and information technology 

Getting started

Review team and advisory group
Information management

Gathering and describing research

Searching for studies
Describing studies

Appraising and synthesising

Quality and relevance appraisal
Synthesising study findings
Drawing conclusions and making recommendations

Using research evidence

 

For a useful overview of systematic reviews, please see: 

Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, 304 pages. ISBN: 9781849201810

The book contains information on many of the areas mentioned in the following list.

Also see: Gough D, Thomas J, Oliver S (2012) Clarifying differences between review designs and methods. Systematic Reviews, 1: 28 

and

Gough D, Oliver S, Thomas J (2013) Learning from Research: Systematic Reviews for Informing Policy Decisions: A Quick Guide. A paper for the Alliance for Useful Evidence. London: Nesta.

Approaches to reviewing

User involvement

Gough D (2007) Giving voice: evidence-informed policy and practice as a democratizing process. InReiss M, DePalma R, Atkinson E (eds) Marginality and difference In education and beyond. London: Trentham Books.

Gough D (2006) User led research synthesis: a participative approach to driving research agendas. Presented at: Sixth International Campbell Colloquium, Los Angeles, 22–24 February.

Harden A, Thomas J, Kavanagh J (2006) Doing systematic reviews for policy-makers: lessons from three systematic reviews. Presented at: Sixth Annual Campbell Collaboration Colloquium, Los Angeles, California, USA, 22-24 February.

Harden A, Oliver S (2001) Who’s listening? Systematically reviewing for ethics and empowerment. In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 123-137.

Jamal F, Langford R, Daniels P, Thomas J, Harden A, Bonell C (2014) Consulting with young people to inform systematic reviews: an example from a review on the effects of schools on health. Health Expectations [online first 3 December 2014] DOI: 10.1111/hex.12312

Oliver S, Clarke-Jones L, Rees R, Milne R, Buchanan P, Gabbay J, Gyte G, Oakley A, Stein K (2004) Involving consumers in research and development agenda setting for the NHS: developing an evidence-based approach. Health Technology Assessment 8 (15) 1-148.

Oliver S, Thomas J, Harden A, Oakley A (2006) Accumulating evidence to bring policy, practice and research together. In: Killoran A, Swann C, Kelly M (eds) Public health evidence: tackling health inequalities. London: Open University Press, pages 125 –140.

Rees R, Oliver S (2012) Stakeholder perspectives and participation in systematic reviews. In: Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 17-35. ISBN: 9781849201810.

Rees R, Oliver S (2007) User Involvement in systematic reviews: an example from health promotion. In Coren E (ed) Collection of examples of service user and carer participation in systematic reviews. London: Social Care Institute for Excellence

Different types of review

Boaz A, Ashby D, Denyer D, Egan M, Harden A, Jones DR, Pawson R, Tranfield D (2006) A multitude of syntheses: a comparison of five approaches from diverse policy fields. Evidence and Policy 2: 479-502.

Brunton V, Thomas J, Harden A, Rees R, Kavanagh J, Oliver S, Shepherd J, Oakley A (2005) Promoting physical activity amongst children outside of physical education classes: a systematic review integrating intervention studies and qualitative studies. Health Education Journal 64: 323-338.

Caird J, Sutcliffe K, Kwan I, Dickson K, Thomas J (2014) Mediating policy-relevant evidence at speed: are systematic reviews of systematic reviews a useful approach? Evidence & Policy 11:81-97. dx.doi.org/10.1332/174426514X13988609036850

Elliott JH, Turner T, Clavisi O, Thomas J, Higgins JPT, Mavergames C, Gruen RL (2014) Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap. PLoS Medicine 11(2): e1001603. doi:10.1371/journal.pmed.1001603

Gough, D, Martin S, and France J (2012) The literature on meta evaluation. In Gough, D, et al. Meta-evaluation of the Impact and Legacy of the London 2012 Olympic Games and Paralympic Games. Developing Methods Paper Interim Report.

Gough D, Thomas J, Oliver S (2012) Clarifying differences between review designs and methods. Systematic Reviews, 1: 28

Gough D, Thomas J (2012) Commonality and diversity in reviews. In: Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 35-66. ISBN: 9781849201810.

Gough D (2007) Typology of evidence reviews (I. Questions, evidence and methods; II. Breadth and depth; III. Methodological approaches; IV. Quality and relevance appraisal; V. Communication, interpretation and application. Series of six posters presented at: National Centre for Research Methods meeting, Manchester, January.

Harden A (2006) Extending the boundaries of systematic reviews to integrate different types of study: examples of methods developed within reviews on young people's health. In J Popay (ed) Moving beyond effectiveness in evidence synthesis. London: National Institute for Health and Clinical Excellence, pages 31-40.

Harden A, Oakley A, Brunton G, Fletcher A (2005) Integrating ‘qualitative’ studies and trials in reviews: reflections from reviews about teenage pregnancy, parenthood and social exclusion. Paper presented at: XIII Cochrane Colloquium, Melbourne, Australia, 22-26 October.

Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley (2004) Applying systematic review methods to studies of people’s views: an example from public health research. Journal of Epidemiology and Community Health 58: 794–800.

Harden A, Oakley A, Oliver S (2001) Peer-delivered health promotion for young people: a systematic review of different study designs. Health Education Journal 60: 339-353.

Lorenc T, Brunton G, Oliver S, Oliver K, Oakley A (2008) Attitudes to walking and cycling among children, young people and parents: a systematic review. Journal of Epidemiology and Community Health 62: 852-857; doi:10.1136/jech.2007.070250.

Oliver S, Harden A, Rees R, Shepherd J, Brunton G, Garcia J, Oakley A (2005) An emerging framework for integrating different types of evidence in systematic reviews for public policy. Evaluation 11: 428-466.

Oliver S, Peersman G (eds) (2001) Using research for effective health promotion. Buckingham: Open University Press.

Rees R, Kavanagh J, Harden A, Shepherd J, Brunton G, Oliver S, Oakley A (2006) Young people and physical activity: a systematic review matching their views to effective interventions. Health Education Research 21 (6) :806-825.

Shepherd J, Harden A, Rees R, Brunton G, Garcia J, Oliver S, Oakley A (2006) Young people and healthy eating: a systematic review of research on barriers and facilitators. Health Education Research 21: 239-257.

Thomas J, Newman M, Oliver S (2013) Rapid evidence assessments of research to inform social policy: taking stock and moving forward. Evidence and Policy, 9(1): 5-27

Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J (2004) Integrating qualitative research with trials in systematic reviews: an example from public health. British Medical Journal 328: 1010-1012.

Thomas J, Harden A (2003) Practical systems for systematic reviews of research to inform policy and practice in education. In Anderson L, Bennett ND (eds) Evidence-informed policy and practice in educational leadership and management: applications and controversies. London: Paul Chapman Publishing, pages 39-54.

Methodological and other challenges

Gough DA (2005) Research for practice in child neglect. In Taylor J, Daniel B (eds) Neglect: practice issues for health and social care. London: Jessica Kingsley, pages 43-56.

Gough DA (2004) Systematic research synthesis to inform the development of policy and practice in education. In Thomas, G, Pring R (eds)  Evidence-based Practice. Buckingham: Open University Press, pages 44-62.

Harden A (2001) The fine details: conducting a systematic review: In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 111-122.

Harden A, Thomas J (2005) Methodological issues in combining diverse study types in systematic reviews. International Journal of Social Research Methods 8: 257-271.

Jackson N, Waters E, Anderson L, Bailie R, Brunton G, Hawe P, Kristjansonn E, Naccarella L, Norris S, Oliver S, Petticrew M, Pienaar E, Popay J, Roberts H, Rogers W, Shepherd J, Sowden A, Thomas H (2005) Criteria for the systematic review of health promotion and public health interventions. Health Promotion International 20: 367-374.

Jackson N, Waters E, Anderson L, Bailie R, Hawe P, Naccarella L, Norris S, Oliver S, Petticrew M, Popay J, Roberts H, Rogers W, Shepherd J, Sowden A, Thomas H (2004) The challenges of systematically reviewing public health interventions. Journal of Public Health 26 (3): 303–307.

Oakley A (2006) Resistances to new technologies of evaluation: education research in the UK as a case study. Evidence and Policy 2 (1): 61-86.

Oakley A, Gough D, Oliver S, Thomas J (2005) The politics of evidence and methodology: lessons from the EPPI-Centre. Evidence & Policy 1 (1) 5–31.

Oakley A (2003) Research evidence, knowledge management and educational practice: early lessons from a systematic approach. London Review of Education 1(1): 21-33.

Oakley A (2001) Evaluating health promotion: methodological diversity. In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 16-31.

Oakley A (2001) Evidence-informed policy and practice: challenges for social science. Manchester: Manchester Statistical Society Papers.

Oakley A (2001) Making evidence-based practice educational: a rejoinder to John Elliott. British Educational Research Journal 27 (5): 575-576.

Oakley A (2000) Experimentation in social science: the case of health promotion. Social Sciences in Health 4: 73-89.

Oakley A (2000) Experiments in knowing: gender and method in the social sciences. Cambridge: Polity Press.

Oakley A (2000) Paradigm wars: some thoughts on a personal and public trajectory. International Journal of Social Research Methodology: Theory and Practice 2: 247-254.

Oliver S, Bagnall AM, Thomas J, Shepherd J, Sowden A, White I, Dinnes J, Rees R, Colquitt J, Oliver K, Garrett Z (2010) RCTs for policy interventions? A review of reviews and meta-regression. Health Technology Assessment 14(16).

Peersman G, Oakley A, Oliver S (1999) Evidence-based health promotion? Some methodological challenges. Journal of the Institute of Health Education 37 (2): 59-64.

Shemilt I, Thomas J, Morciano M (2010) A web-based tool for adjusting costs to a specific target currency and price year. Evidence and Policy 6 (1): 51-59.

Shepherd J, Harden A (2003) How useful are systematic reviews of the effectiveness of sexual health promotion? Exploring the methodological challenges. In Imrie J, Stephenson J, Bonell C (rds) Effective sexual health interventions: issues in experimental evaluation. Oxford: Oxford University Press.

Thomas J, Harden A (2008) Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology 8: 45. doi:10.1186/1471-2288-8-45

Woodman J, Thomas J, Dickson K (2012) How explicable are differences between reviews that appear to address a similar research question? A review of reviews of physical activity interventions. Systematic Reviews. 1: 37.
 

Text mining, automation and information technology

Miwa M, Thomas J, O’Mara-Eves A, Ananiadou S (2014) Reducing systematic review workload through certainty-based screening. Journal of Biomedical Informatics. 51: 242–253

O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S (2015) Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic Reviews 4:5. doi:10.1186/2046-4053-4-5

Shemilt I, Simon A, Hollands G, Marteau TM, Ogilvie D, O’Mara-Eves A, Kelly MP, Thomas J (2014) Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Research Synthesis Methods.5(1): 31-49 [online first 23 August 2013]

Stansfield C, Thomas J, Kavanagh J (2013) ‘Clustering’ documents automatically to support scoping reviews of research: a case study’. Research Synthesis Methods. [online first. 3 July]

Thomas J (2013) Diffusion of innovation in systematic review methodology: why is study selection not yet assisted by automation? OA Evidence-Based Medicine. Oct 21;1(2):12

Thomas J, McNaught J, Ananiadou S (2011) Applications of text mining within systematic reviews. Research Synthesis Methods. 2(1): 1-14.

Thomas J, Brunton J, Graziosi S (2010) EPPI-Reviewer 4.0: software for research synthesis. EPPI-Centre Software. London: Social Science Research Unit, Institute of Education.
 

Getting started 

Review team and advisory group

Harden A, Thomas J, Kavanagh J (2006) Doing systematic reviews for policy-makers: lessons from three systematic reviews. Presented at: Sixth Annual Campbell Collaboration Colloquium, Los Angeles, California, USA, 22-24 February.

Harden A (2001) The fine details: conducting a systematic review: In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 111-122.

Oliver S, Dickson K, Newman M (2012) Getting started with a review. In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 66-83. ISBN: 9781849201810.

Oliver S (2001) Making research more useful: integrating different perspectives and different methods. In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 167-179.

Rees R, Oliver S (2007) User Involvement in systematic reviews: an example from health promotion. In Coren E (ed) Collection of examples of service user and carer participation in systematic reviews. London: Social Care Institute for Excellence.

Information management

Brunton G, Thomas J (2012) Information management in reviews: In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 83-107. ISBN: 9781849201810.

Gathering and describing research 

Searching for studies

Brunton G, Stansfield C, Thomas J (2012) In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 107-135. ISBN: 9781849201810.

Harden A (2001) Finding research evidence: systematic searching.In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 47-68.

Harden A (2001) The fine details: conducting a systematic review:In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 111-122.

Harden A, Peersman G, Oliver S, Oakley A (1999) Identifying relevant primary research on electronic databases to inform decision-making in health promotion: the case of sexual health promotion, Health Education Journal 58 (3): 290-301.

O'Mara-Eves A, Brunton G, McDaid D, Kavanagh J, Oliver S, Thomas J (2014), Techniques for identifying cross-disciplinary and ‘hard-to-detect’ evidence for systematic review. Research Synthesis Methods.5(1): 50-59 [online first 23 August 2013]

Schucan Bird K; Tripney J (2011) Systematic literature searching in policy relevant, inter-disciplinary reviews: an example from culture and sport. Research Synthesis Methods. 2(3): 163-173.

Stansfield C, Kavanagh J, Rees R, Gomersall A, Thomas J (2012) The selection of search sources influences the findings of a systematic review of people's views: a case study in public health. BMC Medical Research Methodology, 12:55

Woodman JL, Harden A, Thomas J, Brunton J, Kavanagh J, Stansfield C (2010) Searching for systematic reviews of the effects of social and environmental interventions: a case study of children and obesity. Journal of the Medical Library Association. 98 (2): 140-146

Describing studies 

Oliver S, Sutcliffe K (2012) Describing and analysing studies. In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 135-153. ISBN: 9781849201810.

Appraising and synthesising the data 

Quality and relevance appraisal 

Gough D (2007) Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. Research Papers in Educatio, 22 (2): 213-228.

Harden A, Gough D (2012) Quality and relevance appraisal. In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 153-179. ISBN: 9781849201810.

Harden A (2007) The quality of qualitative evidence: a review of assessment tools. Paper presented at The Seventh Annual International Campbell Colloquium, May 14-16, London, England.

Harden A (2006) Quality assessment, the research synthesis matrix, and question-driven reviews. Invited paper presented at: ESRC Research Methods Festival, St Catherine’s College Oxford, 19 July.

Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A (2004) Applying systematic review methods to studies of people’s views: an example from public health. Journal of Epidemiology and Community Health 58: 794-800.

Harden A (2001) The fine details: conducting a systematic review: In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 111-122.

Synthesising study findings 

Barnett-Page E, Thomas J (2009) Methods for the synthesis of qualitative research: a critical review BMC Medical Research Methodology 9: 59. doi:10.1186/1471-2288-9-59

Gough D, (2013) Meta-narrative and realist reviews: guidance, rules, publication standards and quality appraisal. BMC Medicine, 11:22

Harden A, Thomas J (2005) Methodological issues in combining diverse study types in systematic reviews. International Journal of Social Research Methods 8: 257-271.

Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A (2004) Applying systematic review methods to studies of people’s views: an example from public health. Journal of Epidemiology and Community Health 58: 794-800.

Thomas J, O’Mara-Eves A, Brunton G (2014) Using Qualitative Comparative Analysis (QCA) in systematic reviews of complex interventions: a worked example. Systematic Reviews. 3: 67

Thomas J, Harden A, Newman M (2012) Synthesis: Combining results systematically and appropriately. In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 179-227. ISBN: 9781849201810.

Thomas J, Harden A (2008) Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology 8: 45.

Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R. Brunton G. Kavanagh J (2004) Integrating qualitative research with trials in systematic reviews: an example from public health. British Medical Journal 328: 1010-1012.

Drawing conclusions and making recommendations 

Bonell C, Oakley A, Hargreaves J, Strange V, Rees R (2006) Assessment of generalisability in trials of health interventions: suggested framework and systematic review. British Medical Journal 333: 346-349.

Oliver S, Peersman G, Harden A, Oakley A (1999) Discrepancies in findings from effectiveness reviews: the case of health promotion for older people. Health Education Journal 58: 78-90.

Peersman G, Harden A, Oliver S, Oakley A (1999) Effectiveness reviews in health promotion: different methods, different recommendations. Health Education Journal 58: 192-202.

Shepherd J, Harden A, Rees R, Brunton G, Garcia J, Oliver S, Oakley A (2006) Young people and healthy eating: a systematic review of research on barriers and facilitators. Health Education Research 21: 239-257.

Using research evidence 

Communication, interpretation and application

Gough D. (2013) England.  In: Levin B, Qi J, Edelstein H, Sohn J (eds.). The Impact of Research in Education: An International Perspective.  Bristol: Policy Press.

Gough D. Tripney, J., Kenny, C., Buk-Berge., E (2011) Evidence Informed Policy in Education in Europe: EIPEE final project report. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London. ISBN: 978-1-907345-15-9.

Gough D (2011) User-led Reviews of Research Knowledge: Enhancing Relevance and Reception. In Banister E et al Knowledge Translation in Context: University-Community, Policy and Indigenous Approaches. Toronto: University of Toronto Press. ISBN 978-1-4426-4179-2

Gough D (2006) User led research synthesis: a participative approach to driving research agendas. Presented at: Sixth International Campbell Colloquium, Los Angeles, 22–24 February.

Harden A, Thomas J, Kavanagh J (2006) Doing systematic reviews for policy-makers: lessons from three systematic reviews. Presented at: Sixth Annual Campbell Collaboration Colloquium, Los Angeles, California, USA, 22-24 February.

Harden A (2001) The fine details: conducting a systematic review. In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 111-122.

Oliver S,  Nicholas A, Oakley A (1996) PHASE: promoting health after sifting the evidence: Workshop report. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.

Stewart R, Oliver S (2012) Making a difference with systematic reviews. In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 227-245. ISBN: 9781849201810.

Stewart R, Oliver S. (2006) Reviewing the potential for critical appraisal training to cater for professional practice. Medical Teacher 28(2): e74-e79.

Stewart R, Hargreaves K, Oliver S (2005) Evidence informed policy making for health communication. Health Education Journal 64 (2): 120-128.

Stewart R, Ellison G (2002) Access to educational research for HIV prevention in southern Africa. EID Review 6: 5-6.

  
Copyright 2006-2009 Social Science Research Unit, Institute of Education :: Privacy Statement :: Terms Of Use :: Site Map :: Login
EPPI-Centre::Evidence Library::Research Use::Learning::Methods & Tools::Databases