PublicationsMethods references
Publications on systematic review / evidence synthesis methodology

Approaches to reviewing

User involvement
Different types of review
Methodological and other challenges
Text mining, automation and information technology 

 

Getting started

Review team and advisory group
Information management

 

Gathering and describing research

Searching for studies
Describing studies

 

Appraising and synthesising

Quality and relevance appraisal

Synthesising study findings

Drawing conclusions and making recommendations

 

Using research evidence

Communication, interpretation and application

 

For a useful overview of systematic reviews, please see:
Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, 304 pages. ISBN: 9781849201810

The book contains information on many of the areas mentioned in the following list.

Also see: Gough D, Thomas J, Oliver S (2012) Clarifying differences between review designs and methods. Systematic Reviews, 1: 28
and
Gough D, Oliver S, Thomas J (2013) Learning from Research: Systematic Reviews for Informing Policy Decisions: A Quick Guide. A paper for the Alliance for Useful Evidence. London: Nesta.

Approaches to reviewing

User involvement

Gough D (2007) Giving voice: evidence-informed policy and practice as a democratizing process. In Reiss M, DePalma R, Atkinson E (eds) London: Trentham Books. Marginality and difference In education and beyond , Los Angeles, California, USA, 22-24 February. Presented at: Sixth Annual Campbell Collaboration Colloquium Doing systematic reviews for policy-makers: lessons from three systematic reviews.

Gough D (2006) User led research synthesis: a participative approach to driving research agendas Los Angeles, 22–24 February. Presented at: Sixth International Campbell Colloquium

Gough D (2006) Buckingham: Open University Press, pages 123-137.. Using research for effective health promotion Who’s listening? Systematically reviewing for ethics and empowerment. In Oliver S, Peersman G (eds) Harden A, Oliver S (2001) Health Expectations [online first 3 December 2014] DOI: 10.1111/hex.12312 Consulting with young people to inform systematic reviews: an example from a review on the effects of schools on health.

Oliver S, Clarke-Jones L, Rees R, Milne R, Buchanan P, Gabbay J, Gyte G, Oakley A, Stein K (2004) Involving consumers in research and development agenda setting for the NHS: developing an evidence-based approach Health Technology Assessment 8 (15) 1-148

Oliver S, Clarke-Jones L, Rees R, Milne R, Buchanan P, Gabbay J, Gyte G, Oakley A, Stein K (2004)  Research synthesis for tackling health inequalities: lessons from methods developed within systematic reviews with a focus on marginalised groups. London: Open University Press, pages 125 –140. In: Killoran A, Swann C, Kelly M (eds) Public health evidence: tackling health inequalities Accumulating evidence to bring policy, practice and research together.

Oliver S, Thomas J, Harden A, Oakley A (2006) Stakeholder perspectives and participation in systematic reviews. In: Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews London: Sage Publications Ltd, pages 17-35. ISBN: 9781849201810.

Rees R, Oliver S (2012) User Involvement in systematic reviews: an example from health promotion. In Coren E (ed) Collection of examples of service user and carer participation in systematic reviews. London: Social Care Institute for Excellence.

Different types of review


Boaz A, Ashby D, Denyer D, Egan M, Harden A, Jones DR, Pawson R, Tranfield D (2006) A multitude of syntheses: a comparison of five approaches from diverse policy fields. 2: 479-502. Evidence and Policy. Health Education Journal Promoting physical activity amongst children outside of physical education classes: a systematic review integrating intervention studies and qualitative studies. Judgment, Resources, and Complexity: A Qualitative Study of the Experiences of Systematic Reviewers of Health Promotion. Evaluation & the Health Professions June 1, 2013 36: 247-267

Caird J, Sutcliffe K, Kwan I, Dickson K, Thomas J (2015) Mediating policy-relevant evidence at speed: are systematic reviews of systematic reviews a useful approach? Evidence & Policy: A Journal of Research, Debate and Practice, Volume 11, Number 1, January 2015, pp. 81-97(17)

Elliott JH, Turner T, Clavisi O, Thomas J, Higgins JPT, Mavergames C, Gruen RL (2014) Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap. PLOS Medicine: http://dx.doi.org/10.1371/journal.pmed.1001603

Gough D, Thomas J (2016) Systematic reviews of research in education: aims, myths and multiple methods. Review of Education. 4:1, pages 84–102

Gough, D, Martin S, Bovaird T and France J (2014) Meta-evaluation of the Impact and Legacy of the London 2012 Olympic Games and Paralympic Games. Developing Methods Paper. Interim Report.

Gough D, Thomas J, Oliver S (2012) Clarifying differences between review designs and methods. Systematic Reviews, 1: 28.

Gough D, Thomas J, Oliver S (2012) Commonality and diversity in reviews. In Gough D, Oliver S, Thomas T: An Introduction to Systematic Reviews. Sage Publications Ltd, pages 35-66. ISBN: 9781849201810.

Gough D, Thomas J (2012) Manchester, January. Series of six posters presented at: National Centre for Research Methods meeting. V. Communication, interpretation and application; IV. Quality and relevance appraisal; III. Methodological approaches; II. Breadth and depth; I. Questions, evidence and methods (Typology of evidence reviews)

Harden A (2006) Extending the boundaries of systematic reviews to integrate different types of study: examples of
methods developed within reviews on young people’s health. In: Popay J (ed): Moving Beyond Effectiveness in Evidence Synthesis: methodological issues in the synthesis of diverse sources of evidence. National Institute of Health and Clinical Excellence, London, pp. 31-40. ISBN 1846291135

Harden A, Oakley A, Brunton G, Fletcher A (2005) Integrating ‘qualitative’ studies and trials in reviews: reflections from reviews about teenage pregnancy, parenthood and social exclusion. Paper presented at: XIII Cochrane Colloquium, Melbourne, Australia, 22-26 October.  

Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley (2004) Applying systematic review methods to studies of people’s views: an example from public health research. Journal of Epidemiology and Community Health 58: 794–800.

Harden A, Oakley A, Oliver S (2001) Peer-delivered health promotion for young people: a systematic review of different study designs. Health Education Journal 60: 339-353.

Lorenc T, Brunton G, Oliver S, Oliver K, Oakley A (2008) Attitudes to walking and cycling among children, young people and parents: a systematic review. Journal of Epidemiology and Community Health 62: 852-857; doi:10.1136/jech.2007.070250.

Oliver S, Harden A, Rees R, Shepherd J, Brunton G, Garcia J, Oakley A (2005) An emerging framework for integrating different types of evidence in systematic reviews for public policy. Evaluation 11: 428-466.

Oliver S, Peersman G (eds) (2001) Using research for effective health promotion. Buckingham: Open University Press.

Popay (2006) Moving beyond effectiveness: methodological issues in the synthesis of diverse sources of evidence. London : National Institute for Health and Clinical Excellence, 2006.

Rees R, Kavanagh J, Harden A, Shepherd J, Brunton G, Oliver S, Oakley A (2006) Young people and physical activity: a systematic review matching their views to effective interventions. Health Education Research 21 (6):806-825.

Shepherd J, Harden A, Rees R, Brunton G, Garcia J, Oliver S, Oakley A (2006) Young people and healthy eating: a systematic review of research on barriers and facilitators. Health Education Research 21: 239-257.

Thomas J, Newman M, Oliver S (2013) Rapid evidence assessments of research to inform social policy: taking stock and moving forward. Evidence and Policy, 9(1): 5-27

Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J (2004) Integrating qualitative research with trials in systematic reviews: an example from public health. British Medical Journal 328: 1010-1012.

Thomas J, Harden A (2003) Practical systems for systematic reviews of research to inform policy and practice in education. In Anderson L, Bennett ND (eds) Evidence-informed policy and practice in educational leadership and management: applications and controversies. London: Paul Chapman Publishing, pages 39-54.

Methodological and other challenges

Gough DA (2005) Research for practice in child neglect. In Taylor J, Daniel B (eds) Neglect: practice issues for health and social care. London: Jessica Kingsley, pages 43-56.

Gough DA (2004) Systematic research synthesis to inform the development of policy and practice in education. In Thomas, G, Pring R (eds)  Evidence-based Practice. Buckingham: Open University Press, pages 44-62.

Harden A (2001) The fine details: conducting a systematic review: In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 111-122.

Harden A, Thomas J (2005) Methodological issues in combining diverse study types in systematic reviews. International Journal of Social Research Methods 8: 257-271.

Jackson N, Waters E, Anderson L, Bailie R, Brunton G, Hawe P, Kristjansonn E, Naccarella L, Norris S, Oliver S, Petticrew M, Pienaar E, Popay J, Roberts H, Rogers W, Shepherd J, Sowden A, Thomas H (2005) Criteria for the systematic review of health promotion and public health interventions. Health Promotion International 20: 367-374.

Jackson N, Waters E, Anderson L, Bailie R, Hawe P, Naccarella L, Norris S, Oliver S, Petticrew M, Popay J, Roberts H, Rogers W, Shepherd J, Sowden A, Thomas H (2004) The challenges of systematically reviewing public health interventions. Journal of Public Health 26 (3): 303–307.

Oakley A (2006) Resistances to new technologies of evaluation: education research in the UK as a case study. Evidence and Policy 2 (1): 61-86.

Oakley A, Gough D, Oliver S, Thomas J (2005) The politics of evidence and methodology: lessons from the EPPI-Centre. Evidence & Policy 1 (1) 5–31.

Oakley A (2003) Research evidence, knowledge management and educational practice: early lessons from a systematic approach. London Review of Education 1(1): 21-33.

Oakley A (2001) Evaluating health promotion: methodological diversity. In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 16-31.

Oakley A (2001) Evidence-informed policy and practice: challenges for social science. Manchester: Manchester Statistical Society Papers.

Oakley A (2001) Making evidence-based practice educational: a rejoinder to John Elliott. British Educational Research Journal 27 (5): 575-576.

Oakley A (2000) Experimentation in social science: the case of health promotion. Social Sciences in Health 4: 73-89.

Oakley A (2000) Experiments in knowing: gender and method in the social sciences. Cambridge: Polity Press.

Oakley A (2000) Paradigm wars: some thoughts on a personal and public trajectory. International Journal of Social Research Methodology: Theory and Practice 2: 247-254.

Oliver S, Bagnall AM, Thomas J, Shepherd J, Sowden A, White I, Dinnes J, Rees R, Colquitt J, Oliver K, Garrett Z (2010) RCTs for policy interventions? A review of reviews and meta-regression. Health Technology Assessment 14(16).

Peersman G, Oakley A, Oliver S (1999) Evidence-based health promotion? Some methodological challenges. Journal of the Institute of Health Education 37 (2): 59-64.

Shemilt I, Thomas J, Morciano M (2010) A web-based tool for adjusting costs to a specific target currency and price year. Evidence and Policy 6 (1): 51-59.

Shepherd J, Harden A (2003) How useful are systematic reviews of the effectiveness of sexual health promotion? Exploring the methodological challenges. In Imrie J, Stephenson J, Bonell C (rds) Effective sexual health interventions: issues in experimental evaluation. Oxford: Oxford University Press.

Thomas J, Harden A (2008) Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology 8: 45. doi:10.1186/1471-2288-8-45

Woodman J, Thomas J, Dickson K (2012) How explicable are differences between reviews that appear to address a similar research question? A review of reviews of physical activity interventions. Systematic Reviews. 1: 37.  

Text mining, automation and information technology

Miwa M, Thomas J, O’Mara-Eves A, Ananiadou S (2014) Reducing systematic review workload through certainty-based screening. Journal of Biomedical Informatics. 51: 242–253

O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S (2015) Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic Reviews 4:5. doi:10.1186/2046-4053-4-5

Shemilt I, Simon A, Hollands G, Marteau TM, Ogilvie D, O’Mara-Eves A, Kelly MP, Thomas J (2014) Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Research Synthesis Methods.5(1): 31-49 [online first 23 August 2013]

Stansfield C, Thomas J, Kavanagh J (2013) ‘Clustering’ documents automatically to support scoping reviews of research: a case study’. Research Synthesis Methods. [online first. 3 July]

Thomas J (2013) Diffusion of innovation in systematic review methodology: why is study selection not yet assisted by automation? OA Evidence-Based Medicine. Oct 21;1(2):12

Thomas J, McNaught J, Ananiadou S (2011) Applications of text mining within systematic reviews. Research Synthesis Methods. 2(1): 1-14.

Thomas J, Brunton J, Graziosi S (2010) EPPI-Reviewer 4.0: software for research synthesis. EPPI-Centre Software. London: Social Science Research Unit, Institute of Education.   

Getting started  

Review team and advisory group

Harden A, Thomas J, Kavanagh J (2006) Doing systematic reviews for policy-makers: lessons from three systematic reviews. Presented at: Sixth Annual Campbell Collaboration Colloquium, Los Angeles, California, USA, 22-24 February. Harden A (2001) The fine details: conducting a systematic review: In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 111-122.

Kneale D, Thomas J, Harris K (2015) Developing and Optimising the Use of Logic Models in Systematic Reviews: Exploring Practice and Good Practice in the Use of Programme Theory in Reviews. PLoS One, 10(11).

Oliver S, Dickson K, Newman M (2012) Getting started with a review. In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 66-83. ISBN: 9781849201810.

Oliver S (2001) Making research more useful: integrating different perspectives and different methods. In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 167-179. Rees R, Oliver S (2007) User Involvement in systematic reviews: an example from health promotion. In Coren E (ed) Collection of examples of service user and carer participation in systematic reviews. London: Social Care Institute for Excellence.

Information management

Brunton G, Thomas J (2012) Information management in reviews: In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 83-107. ISBN: 9781849201810.

Gathering and describing research  

Searching for studies

Brunton G, Stansfield C, Thomas J (2012) In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 107-135. ISBN: 9781849201810.

Harden A (2001) Finding research evidence: systematic searching.In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 47-68.

Harden A (2001) The fine details: conducting a systematic review:In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 111-122.

Harden A, Peersman G, Oliver S, Oakley A (1999) Identifying relevant primary research on electronic databases to inform decision-making in health promotion: the case of sexual health promotion, Health Education Journal 58 (3): 290-301.

O'Mara-Eves A, Brunton G, McDaid D, Kavanagh J, Oliver S, Thomas J (2014), Techniques for identifying cross-disciplinary and ‘hard-to-detect’ evidence for systematic review. Research Synthesis Methods.5(1): 50-59 [online first 23 August 2013]

Schucan Bird K; Tripney J (2011) Systematic literature searching in policy relevant, inter-disciplinary reviews: an example from culture and sport. Research Synthesis Methods. 2(3): 163-173.

Stansfield C, Kavanagh J, Rees R, Gomersall A, Thomas J (2012) The selection of search sources influences the findings of a systematic review of people's views: a case study in public health. BMC Medical Research Methodology, 12:55

Woodman JL, Harden A, Thomas J, Brunton J, Kavanagh J, Stansfield C (2010) Searching for systematic reviews of the effects of social and environmental interventions: a case study of children and obesity. Journal of the Medical Library Association. 98 (2): 140-146

Describing studies  

Kneale D, Thomas J, Harris K (2015) Developing and Optimising the Use of Logic Models in Systematic Reviews: Exploring Practice and Good Practice in the Use of Programme Theory in Reviews. PLoS One, 10(11).

Oliver S, Sutcliffe K (2012) Describing and analysing studies. In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 135-153. ISBN: 9781849201810.

Appraising and synthesising the data

Quality and relevance appraisal  

Gough D (2007) Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. Research Papers in Educatio, 22 (2): 213-228. Harden A, Gough D (2012) Quality and relevance appraisal. In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 153-179. ISBN: 9781849201810.

Harden A (2007) The quality of qualitative evidence: a review of assessment tools. Paper presented at The Seventh Annual International Campbell Colloquium, May 14-16, London, England.

Harden A (2006) Quality assessment, the research synthesis matrix, and question-driven reviews. Invited paper presented at: ESRC Research Methods Festival, St Catherine’s College Oxford, 19 July.

Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A (2004) Applying systematic review methods to studies of people’s views: an example from public health. Journal of Epidemiology and Community Health 58: 794-800.

Harden A (2001) The fine details: conducting a systematic review: In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 111-122.


Synthesising study findings

Barnett-Page E, Thomas J (2009) Methods for the synthesis of qualitative research: a critical review.  BMC Medical Research Methodology 9: 59. doi:10.1186/1471-2288-9-59

Gough D, (2013) Meta-narrative and realist reviews: guidance, rules, publication standards and quality appraisal. BMC Medicine, 11:22 Harden A, Thomas J (2005) Methodological issues in combining diverse study types in systematic reviews. International Journal of Social Research Methods 8: 257-271.

Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A (2004) Applying systematic review methods to studies of people’s views: an example from public health. Journal of Epidemiology and Community Health 58: 794-800.

O'Mara-Eves, A. and Thomas, J. (2016) Ongoing developments in meta-analytic and quantitative synthesis methods: Broadening the types of research questions that can be addressed. Review of Education, 4: 5–27. doi: 10.1002/rev3.3062

Sutcliffe K, Thomas J, Stokes G, Hinds K, Bangpan M (2015) Intervention Component Analysis (ICA): a pragmatic approach for identifying the critical features of complex interventions. Systematic Reviews 2015, 4:140

Thomas J, O’Mara-Eves A, Brunton G (2014) Using Qualitative Comparative Analysis (QCA) in systematic reviews of complex interventions: a worked example. Systematic Reviews. 3: 67

Thomas J, Harden A, Newman M (2012) Synthesis: Combining results systematically and appropriately. In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 179-227. ISBN: 9781849201810.

Thomas J, Harden A (2008) Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology 8: 45.

Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R. Brunton G. Kavanagh J (2004) Integrating qualitative research with trials in systematic reviews: an example from public health. British Medical Journal 328: 1010-1012.


Drawing conclusions and making recommendations

Bonell C, Oakley A, Hargreaves J, Strange V, Rees R (2006) Assessment of generalisability in trials of health interventions: suggested framework and systematic review. British Medical Journal 333: 346-349.

Oliver S, Peersman G, Harden A, Oakley A (1999) Discrepancies in findings from effectiveness reviews: the case of health promotion for older people. Health Education Journal 58: 78-90.

Peersman G, Harden A, Oliver S, Oakley A (1999) Effectiveness reviews in health promotion: different methods, different recommendations. Health Education Journal 58: 192-202.

Shepherd J, Harden A, Rees R, Brunton G, Garcia J, Oliver S, Oakley A (2006) Young people and healthy eating: a systematic review of research on barriers and facilitators. Health Education Research 21: 239-257.

Using research evidence  

Communication, interpretation and application

Gough D. (2013) England. In: Levin B, Qi J, Edelstein H, Sohn J (eds.). The Impact of Research in Education: An International Perspective.  Bristol: Policy Press.

Gough D. Tripney, J., Kenny, C., Buk-Berge., E (2011) Evidence Informed Policy in Education in Europe: EIPEE final project report. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London. ISBN: 978-1-907345-15-9.

Gough D (2011) User-led Reviews of Research Knowledge: Enhancing Relevance and Reception. In Banister E et al. Knowledge Translation in Context: University-Community, Policy and Indigenous Approaches. Toronto: University of Toronto Press. ISBN 978-1-4426-4179-2

Gough D (2006) User led research synthesis: a participative approach to driving research agendas. Presented at: Sixth International Campbell Colloquium, Los Angeles, 22–24 February.

Harden A, Thomas J, Kavanagh J (2006) Doing systematic reviews for policy-makers: lessons from three systematic reviews. Presented at: Sixth Annual Campbell Collaboration Colloquium, Los Angeles, California, USA, 22-24 February.

Harden A (2001) The fine details: conducting a systematic review. In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 111-122.

Oliver S, Nicholas A, Oakley A (1996) PHASE: promoting health after sifting the evidence: Workshop report. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.

Stewart R, Oliver S (2012) Making a difference with systematic reviews. In Gough, D, Oliver, S, Thomas, J (2012) An Introduction to Systematic Reviews. London: Sage Publications Ltd, pages 227-245. ISBN: 9781849201810.

Stewart R, Oliver S. (2006) Reviewing the potential for critical appraisal training to cater for professional practice. Medical Teacher 28(2): e74-e79.

Stewart R, Hargreaves K, Oliver S (2005) Evidence informed policy making for health communication. Health Education Journal 64 (2): 120-128.

Stewart R, Ellison G (2002) Access to educational research for HIV prevention in southern Africa. EID Review 6: 5-6.

  
Copyright 2015 Social Science Research Unit, UCL Institute of Education :: Privacy Statement :: Terms Of Use :: Site Map :: Login
About us::Research projects::Publications::Research Use::Courses & seminars::Resources::Databases