Contact: Ian Shemilt

Start Date: 1st January 2024

End Date: TBC - Ongoing

Background

Generative large language model- (LLM) based tools are a class of artificial intelligence (AI) tools capable of generating natural language text, images, audio, or other media outputs in response to user- or self-inputs, or 'prompts’). Some well-known generative LLMs are OpenAI's GPT family of models which underpin ChatGPT, Google's Gemini (formerly known as Bard), Llama (Meta), Claude (Anthropic), and Mistral (Mistral AI). These models can generate content - that is, they can perform 'generative decoder operations’ - because generative LLMs can process (‘encode’) - and thereby ‘learn’ to predict (‘decode’) - the syntax, semantics, and patterns of natural language text (or the patterns of images or audio). As such, generative LLMs are essentially only capable of ‘next word prediction’ (or equivalents for images and audio).

Generative LLM-based tools have a range of potential applications in clinical health, public health, or social care. Examples of applications in clinical health care may include: diagnosing medical conditions, predicting disease progression, extracting information from patient records, writing medical reports, and supporting shared decision-making between clinicians and patients. 

Objectives

The overall aim of this project is: 

• To empower key policy and other stakeholders with the enabling knowledge and skills needed to ask relevant questions and make informed judgments about the utility, reliability, and potential risks of generative LLM-based tools, when considering these for potential adoption for specific health and social care applications. 

Methods

We plan to produce two main outputs: 1) a living evidence and gap map (living EGM); and 2) a critical review of evidence claims about identified classes of health and social applications. Further details of the specific objectives, scope and methods of the living EGM and critical review are provided in our protocol (see 'Publications', below).

Publications

Shemilt I, Hollands GJ, Khouja C, Raine G, Kneale D, O’Mara-Eves A, Sutcliffe K, Thomas J (2024). Generative Large Language Model-Based Tools for Health and Social Care Applications: A Living Map and Critical Review (Protocol). London: EPPI Centre, Social Science Research Unit, UCL Institute of Education, University College London. ISBN: 978-1-911605-57-7

Open Science Framework (OSF) (10th June 2024)

Research ethics application form (1st December 2023)

Research ethics approval letter (6th December 2023)

Further information

This work is being undertaken by a team of researchers based at the EPPI Centre under the auspices of the UK NIHR Policy Research Programme Reviews Facility (NIHR PRP Reviews Facility). The NIHR PRP Reviews Facility is a collaboration between the EPPI Centre (UCL), the Centre for Reviews and Dissemination (CRD, University of York) and the Department of Public Health, Environments and Society (PHES, London School of Hygiene and Tropical Medicine).