What is this map?
This living map of research contains bibliographic records of studies about generative large language model- (LLM-) based tools for health and social care applications.
We identify studies using a semi-automated continual evidence surveillance workflow featuring automated searches of a large open access knowledge graph dataset -- containing records of 250 million research articles from across science -- using OpenAlex tools in EPPI Reviewer. See 'About this Map' (.docx) for further details.
The main reason this map focuses on studies of generative LLM-based tools - and does not contain studies of other kinds of (non-generative) artificial intelligence (AI) or machine learning tools - is that the possibility of generative LLM-based tools producing reliable, or accurate, responses is dependent on unique features of their design that the other kinds of tools do not share.
What can this map be used for?
This living map can be used to find, access and reuse published research articles on the performance of generative LLM-based tools for various kinds of health and social care applications and tasks. This includes a series of pre-configured evidence and gap maps (EGMs), which aim to provide a visual overview of the landscape of current research evidence -- highlighting areas where there is more, or less, research evidence within the overall scope of this topic. We have prepared six pre-configured EGMs (see below on this page), or users can configure their own via the web database -- here is a short YouTube video with some tips on 'how'.
What can't this map be used for?
We have not critically appraised studies included in this living map but users should expect their quality to be variable. As such, this map cannot be used to distinguish between better or worse quality research on this topic. Evidence claims for the performance of these tools for specific classes of health or social care applications will be assessed as part of our forthcoming critical review on this topic.
For further information about the methods we have used to identify and classify (code) records of articles (reports) included in this living map (the 'main map' or the 'sector'), please see 'About this Map' (docx). This living document also contains 'results' information, including flows of records through our eligibility screening and coding workflows and the cumulative number of records included in the map.
Main Map
The 'main map' (below) contains studies that include a substantive focus on one or more of the following classes of applications: ‘clinical health care’, ‘public health’ and/ or ‘social care’.
Access the latest version of the 'main map', which will open as an interactive web database: here.
The latest version of the main map is Version 1 - 2nd September 2024 (published 18th October 2024) and it contains 146 fully coded records. The web database includes an 'Introduction' with a section on 'How to use this map'.
.
Each record included in the 'main map' has been classified (coded) according to the following study characteristics (dimensions):
- Application class(es)
- Article type
- Model(s)
- Mode(s) of model(s) use
- Type(s) of task(s)
- Health and social care categories
- Service type(s)
- Population(s)
- Version
As well as opening as an interactive web database (above) users can also open the 'main map' to display its contents in any of the six available pre-configured evidence and gap map (EGM) views:
'Types of task(s)' by 'Application class(es)' by 'Article type' (Pre-configured EGM #1)
'Types of task(s)' by 'Health and social care categories' by 'Article type' (Pre-configured EGM #2)
'Type(s) of task(s)' by 'Service type(s)' by 'Article type' (Pre-configured EGM #3)
'Types of task(s)' by 'Model(s)' by 'Article type' (Pre-configured EGM #4)
'Model(s)' by 'Health and social care categories' by 'Article Type' (Pre-configured EGM #5)
'Model(s)' Models' by 'Article Type' (Pre-configured EGM #6)
Sector
Alongside the 'main map', we also publish a separate 'sector' of this living map containing further eligible records (held back from the 'main map') which are only about specific classes of applications of generative LLM-based tools: ‘medical education, health or social care professional development and training’, ‘answering medical (and related) exam questions/ answering medical questions in general’ and/ or ‘answering patient or service user questions/ patient or service user education’. Access the latest version of this 'sector', which will open as a second, separate interactive web database here.
How to cite this living map
Shemilt I, Hollands GJ, Khouja C, Raine G, Thomas J. (2024). Generative large language models for health and social care applications: a living map of research. London: EPPI Centre.
The published protocol that covers this living map is available here as well as on the main project page. For contact details please see the main project page.
Further information
This work is being undertaken by a team of researchers based at the EPPI Centre under the auspices of the UK NIHR Policy Research Programme Reviews Facility (NIHR PRP Reviews Facility). The NIHR PRP Reviews Facility is a collaboration between the EPPI Centre (UCL), the Centre for Reviews and Dissemination (CRD, University of York) and the Department of Public Health, Environments and Society (PHES, London School of Hygiene and Tropical Medicine).