Publications on systematic review and evidence synthesis methodology
Approaches to reviewing
User involvement
Different types of review
Methodological and other challenges
Text mining, machine learning, automation and information technology
Getting started
Review team and advisory group
Information management
Gathering and describing research
Searching for studies
Describing studies
Appraising and synthesising
Synthesising study findings
Quality and relevance appraisal
Drawing conclusions and making recommendations
Using research evidence
Communication, interpretation and application
For a useful overview of systematic reviews, please see:
Gough, D, Oliver, S, Thomas, J (2017) An Introduction to Systematic Reviews. 2nd edition. London: Sage Publications Ltd, 304 pages. ISBN: 9781849201810
(The book contains information on many of the areas mentioned in the following list.) Also see:
Gough D, Thomas J, Oliver S (2012) Clarifying differences between review designs and methods. Systematic Reviews, 1: 28
Gough D, Thomas J and Oliver S (2019) Clarifying differences between reviews within evidence ecosystems. 8. Systematic Reviews: 1–15.Systematic Reviews
Gough D, Oliver S, Thomas J (2013) A paper for the Alliance for Useful Evidence. London: Nesta. Learning from Research: Systematic Reviews for Informing Policy Decisions: A Quick Guide
. In Reiss M, DePalma R, Atkinson E (eds) London: Trentham Books. Marginality and difference In education and beyond , Los Angeles, California, USA, 22-24 February. Presented at: Sixth Annual Campbell Collaboration Colloquium Doing systematic reviews for policy-makers: lessons from three systematic reviews.Giving voice: evidence-informed policy and practice as a democratizing processGough D (2007)
Los Angeles, 22–24 February. Presented at: Sixth International Campbell ColloquiumUser led research synthesis: a participative approach to driving research agendasGough D (2006)
Gough D (2006) Buckingham: Open University Press, pages 123-137.. Using research for effective health promotion Who’s listening? Systematically reviewing for ethics and empowerment. In Oliver S, Peersman G (eds) Harden A, Oliver S (2001) Health Expectations [online first 3 December 2014] DOI: 10.1111/hex.12312 Consulting with young people to inform systematic reviews: an example from a review on the effects of schools on health.
Health Technology Assessment 8 (15) 1-148Involving consumers in research and development agenda setting for the NHS: developing an evidence-based approachOliver S, Clarke-Jones L, Rees R, Milne R, Buchanan P, Gabbay J, Gyte G, Oakley A, Stein K (2004)
Oliver S, Clarke-Jones L, Rees R, Milne R, Buchanan P, Gabbay J, Gyte G, Oakley A, Stein K (2004) Research synthesis for tackling health inequalities: lessons from methods developed within systematic reviews with a focus on marginalised groups. London: Open University Press, pages 125 –140. In: Killoran A, Swann C, Kelly M (eds) Public health evidence: tackling health inequalities Accumulating evidence to bring policy, practice and research together.
Rees R, Oliver S (2006) Stakeholder perspectives and participation in systematic reviews. In: Gough, D, Oliver, S, Thomas, J (2017) An Introduction to Systematic Reviews London: Sage Publications Ltd, pages 19-42. ISBN: 9781473929432.
Rees R, Oliver S (2012) User Involvement in systematic reviews: an example from health promotion. In Coren E (ed) Collection of examples of service user and carer participation in systematic reviews. London: Social Care Institute for Excellence.
Boaz A, Ashby D, Denyer D, Egan M, Harden A, Jones DR, Pawson R, Tranfield D (2006) A multitude of syntheses: a comparison of five approaches from diverse policy fields. 2: 479-502. Evidence and Policy. Health Education Journal Promoting physical activity amongst children outside of physical education classes: a systematic review integrating intervention studies and qualitative studies. Judgment, Resources, and Complexity: A Qualitative Study of the Experiences of Systematic Reviewers of Health Promotion. Evaluation & the Health Professions June 1, 2013 36: 247-267
Caird J, Sutcliffe K, Kwan I, Dickson K, Thomas J (2015) Mediating policy-relevant evidence at speed: are systematic reviews of systematic reviews a useful approach? Evidence & Policy: A Journal of Research, Debate and Practice, Volume 11, Number 1, January 2015, pp. 81-97(17)
http://dx.doi.org/10.1371/journal.pmed.1001603Elliott JH, Turner T, Clavisi O, Thomas J, Higgins JPT, Mavergames C, Gruen RL (2014) Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap. PLOS Medicine:
Gough D, Thomas J (2016) Systematic reviews of research in education: aims, myths and multiple methods. Review of Education. 4:1, pages 84–102
. Developing Methods Paper. Interim Report.Meta-evaluation of the Impact and Legacy of the London 2012 Olympic Games and Paralympic GamesGough, D, Martin S, Bovaird T and France J (2014)
. Systematic Reviews, 1: 28.Clarifying differences between review designs and methodsGough D, Thomas J, Oliver S (2012)
Gough D, Thomas J (2012) Commonality and diversity in reviews. In: Gough, D, Oliver, S, Thomas, J (2017) An Introduction to Systematic Reviews London: Sage Publications Ltd, pages 43-70. ISBN: 9781473929432
). Series of six posters presented at National Centre for Research Methods meeting, Manchester, January 2007. London: EPPI Centre.V. Communication, interpretation and application; IV. Quality and relevance appraisal; III. Methodological approaches; II..Breadth and depth; I. Questions, evidence and methods (Typology of evidence reviewsGough D (2007)
methods developed within reviews on young people’s health. In: Popay J (ed): Moving Beyond Effectiveness in Evidence Synthesis: methodological issues in the synthesis of diverse sources of evidence. National Institute of Health and Clinical Excellence, London, pp. 31-40. ISBN 1846291135
Harden A (2006) Extending the boundaries of systematic reviews to integrate different types of study: examples of
Harden A, Oakley A, Brunton G, Fletcher A (2005) Integrating ‘qualitative’ studies and trials in reviews: reflections from reviews about teenage pregnancy, parenthood and social exclusion. Paper presented at: XIII Cochrane Colloquium, Melbourne, Australia, 22-26 October.
Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley (2004) Applying systematic review methods to studies of people’s views: an example from public health research. Journal of Epidemiology and Community Health 58: 794–800.
Harden A, Oakley A, Oliver S (2001) Peer-delivered health promotion for young people: a systematic review of different study designs. Health Education Journal 60: 339-353.
Hinds K, Dickson K (2021) 'Realist Synthesis: A critique and an alternative' Journal of Critical Realism
Lorenc T, Brunton G, Oliver S, Oliver K, Oakley A (2008) Attitudes to walking and cycling among children, young people and parents: a systematic review. Journal of Epidemiology and Community Health 62: 852-857; doi:10.1136/jech.2007.070250.
Oliver S, Harden A, Rees R, Shepherd J, Brunton G, Garcia J, Oakley A (2005) An emerging framework for integrating different types of evidence in systematic reviews for public policy. Evaluation 11: 428-466.
Oliver S, Peersman G (eds) (2001) Using research for effective health promotion. Buckingham: Open University Press.
Popay (2006) Moving beyond effectiveness: methodological issues in the synthesis of diverse sources of evidence. London : National Institute for Health and Clinical Excellence, 2006.
Rees R, Kavanagh J, Harden A, Shepherd J, Brunton G, Oliver S, Oakley A (2006) Young people and physical activity: a systematic review matching their views to effective interventions. Health Education Research 21 (6):806-825.
Shepherd J, Harden A, Rees R, Brunton G, Garcia J, Oliver S, Oakley A (2006) Young people and healthy eating: a systematic review of research on barriers and facilitators. Health Education Research 21: 239-257.
Thomas J, Newman M, Oliver S (2013) Rapid evidence assessments of research to inform social policy: taking stock and moving forward. Evidence and Policy, 9(1): 5-27
Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J (2004) Integrating qualitative research with trials in systematic reviews: an example from public health. British Medical Journal 328: 1010-1012.
Thomas J, Harden A (2003) Practical systems for systematic reviews of research to inform policy and practice in education. In Anderson L, Bennett ND (eds) Evidence-informed policy and practice in educational leadership and management: applications and controversies. London: Paul Chapman Publishing, pages 39-54.
8. Systematic Reviews: 1–15.Gough D, Thomas J and Oliver S (2019) Clarifying differences between reviews within evidence ecosystems. Systematic Reviews
Gough DA (2005) Research for practice in child neglect. In Taylor J, Daniel B (eds) Neglect: practice issues for health and social care. London: Jessica Kingsley, pages 43-56.
Gough DA (2004) Systematic research synthesis to inform the development of policy and practice in education. In Thomas, G, Pring R (eds) Evidence-based Practice. Buckingham: Open University Press, pages 44-62.
Harden A (2001) The fine details: conducting a systematic review: In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 111-122.
Harden A, Thomas J (2005) Methodological issues in combining diverse study types in systematic reviews. International Journal of Social Research Methods 8: 257-271.
Jackson N, Waters E, Anderson L, Bailie R, Brunton G, Hawe P, Kristjansonn E, Naccarella L, Norris S, Oliver S, Petticrew M, Pienaar E, Popay J, Roberts H, Rogers W, Shepherd J, Sowden A, Thomas H (2005) Criteria for the systematic review of health promotion and public health interventions. Health Promotion International 20: 367-374.
Jackson N, Waters E, Anderson L, Bailie R, Hawe P, Naccarella L, Norris S, Oliver S, Petticrew M, Popay J, Roberts H, Rogers W, Shepherd J, Sowden A, Thomas H (2004) The challenges of systematically reviewing public health interventions. Journal of Public Health 26 (3): 303–307.
Oakley A (2006) Resistances to new technologies of evaluation: education research in the UK as a case study. Evidence and Policy 2 (1): 61-86.
Oakley A, Gough D, Oliver S, Thomas J (2005) The politics of evidence and methodology: lessons from the EPPI Centre. Evidence & Policy 1 (1) 5–31.
Oakley A (2003) Research evidence, knowledge management and educational practice: early lessons from a systematic approach. London Review of Education 1(1): 21-33.
Oakley A (2001) Evaluating health promotion: methodological diversity. In Oliver S, Peersman G (eds) Using research for effective health promotion. Buckingham: Open University Press, pages 16-31.
Oakley A (2001) Evidence-informed policy and practice: challenges for social science. Manchester: Manchester Statistical Society Papers.
Oakley A (2001) Making evidence-based practice educational: a rejoinder to John Elliott. British Educational Research Journal 27 (5): 575-576.
Oakley A (2000) Experimentation in social science: the case of health promotion. Social Sciences in Health 4: 73-89.
Oakley A (2000) Experiments in knowing: gender and method in the social sciences. Cambridge: Polity Press.
Oakley A (2000) Paradigm wars: some thoughts on a personal and public trajectory. International Journal of Social Research Methodology: Theory and Practice 2: 247-254.
Oliver S, Bagnall AM, Thomas J, Shepherd J, Sowden A, White I, Dinnes J, Rees R, Colquitt J, Oliver K, Garrett Z (2010) RCTs for policy interventions? A review of reviews and meta-regression. Health Technology Assessment 14(16).
Peersman G, Oakley A, Oliver S (1999) Evidence-based health promotion? Some methodological challenges. Journal of the Institute of Health Education 37 (2): 59-64.
Shemilt I, Thomas J, Morciano M (2010) A web-based tool for adjusting costs to a specific target currency and price year. Evidence and Policy 6 (1): 51-59.
Shepherd J, Harden A (2003) How useful are systematic reviews of the effectiveness of sexual health promotion? Exploring the methodological challenges. In Imrie J, Stephenson J, Bonell C (rds) Effective sexual health interventions: issues in experimental evaluation. Oxford: Oxford University Press.
Thomas J, Harden A (2008) Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology 8: 45. doi:10.1186/1471-2288-8-45
Woodman J, Thomas J, Dickson K (2012) How explicable are differences between reviews that appear to address a similar research question? A review of reviews of physical activity interventions. Systematic Reviews. 1: 37.
.10(16):1–10Arno AD, Elliott J, Wallace B, Turner T, Thomas J. (2021) The views of health guideline developers on the use of automation in health evidence synthesis. Systematic Reviews
https://doi.org/10.1038/d41586-021-03690-1 600, 383-385 doi: Elliott J, Lawrence R, Minx JC, Oladapo OT, Ravaud P, Tendal Jeppesen B, Thomas J, Turner T, Vandvik PO, Grimshaw JM (2021) Decision makers need constantly updated evidence synthesis. Nature
8:143O’Connor AM, Tsafnat G, Thomas J, Glasziou P, Gilbert SB, Hutton B (2019) A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? Systematic Reviews
https://doi.org/10.1093/jamia/ocaa163, ocaa163, Marshall IJ, Nye B, Kuiper J, Noel-Storr A, Marshall R, Maclean R, Soboczenski F, Nenkova A, Thomas J, Wallace BC (2020) Trialstreamer: A living, automatically updated database of clinical trial reports, Journal of the American Medical Informatics Association
Miwa M, Thomas J, O’Mara-Eves A, Ananiadou S (2014) Reducing systematic review workload through certainty-based screening. Journal of Biomedical Informatics. 51: 242–253
O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S (2015) Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic Reviews 4:5. doi:10.1186/2046-4053-4-5
, 10:401 (https://doi.org/10.12688/f1000research.51117.1Schmidt L, Olorisade BK, McGuinness LA, Thomas J, Higgins JPT. (2021) Data extraction methods for systematic review (semi)automation: A living systematic review [version 1; peer review: 3 approved]. F1000Research
6:210Shemilt I*, Arno A*, Thomas J*, Lorenc T, Khouja C, Raine G, Sutcliffe K, D'Souza P, Kwan I, Wright K, Sowden A (2021) Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research. Wellcome Open Research:
[Internet]. 23Jun.2021 [cited 30Jun.2021];17(2):11-5Shemilt I, Arno A, Thomas J, Lorenc T, Khouja C, Raine G, Sutcliffe K, D’Souza P, Wright K, Sowden A. (2021) Using automation to produce a ‘living map’ of the COVID-19 research literature. JEAHIL
Shemilt I, Noel-Storr A, Thomas J, Featherstone R, Mavergames C. Machine Learning Reduced Workload for the Cochrane COVID-19 Study Register: Development and Evaluation of the Cochrane COVID-19 Study Classifier, 14 July 2021, PREPRINT (Version 1) available at Research Square [https://doi.org/10.21203/rs.3.rs-689189/v1]
Shemilt I, Simon A, Hollands G, Marteau TM, Ogilvie D, O’Mara-Eves A, Kelly MP, Thomas J (2014) Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Research Synthesis Methods.5(1): 31-49 [online first 23 August 2013]
Stansfield C, Thomas J, Kavanagh J (2013) ‘Clustering’ documents automatically to support scoping reviews of research: a case study’. Research Synthesis Methods. [online first. 3 July]
[online first 7 November 2021] https://doi.org/10.1002/jrsm.1537Stansfield C, Stokes G, Thomas J (2021) Applying machine classifiers to update searches: analysis from two case studies. Research Synthesis Methods
[Internet]. 23Jun.2021 [cited 30Jun.2021];17(2):5Thomas J. (2021) Evidence surveillance during the COVID-19 pandemic using automation and crowdsourcing. JEAHIL
Thomas J, McDonald S, Noel-Storr A, Shemilt I, Elliott J, Mavergames C, Marshall IJ (2020) Machine learning reduces workload with minimal risk of missing studies: development and evaluation of an RCT classifier for Cochrane Reviews. Journal of Clinical Epidemiology
Thomas J, Noel-Storr A, Marshall I, Wallace B, McDonald S, Mavergames C, Glasziou P, Shemilt I, Synnot A, Turner T, Elliott J on behalf of the Living Systematic Review Network (2017) Living Systematic Reviews: 2. Combining Human and Machine Effort. Journal of Clinical Epidemiology