Version 4.15.3.0 is a kind of iceberg update, with one new feature (a new classifier) while otherwise little changes are visible on the surface. It however contains a large amount of new experimental code, which is currently hidden to regular EPPI Reviewer users. Other visible/public changes are limited to bugfixes/improvements and the new-but-hidden functionalities are focussed on enabling medium to large scale evaluations for GPT4-driven coding (more details).

New classifier: PubMed Study Designs

This classifier will attempt to discriminate studies and journal articles by "type", picking from a list of predetermined options such as "Clinical Trial", "RCT", "Case Control Study", "Case Report", but also "Meta Analysis", "Guideline", "Editorial" and more. More details on how this classifier was trained and validated are here.

Enhancement: Items Lists

Having the ability to retrieve Item Lists based on extremely variable criteria is one of the key functionalities of EPPI Reviewer, it however poses a technical challenge. In order to obtain these lists, sometimes it is necessary to read and evaluate vast amounts of data in the database, which is why, especially when opening large reviews, the default list might take a long time to load, and/or time out. In fact, time out errors can happen for different types of Item Lists and the underlying queries typically are the ones for which we spend the most time optimising performance. EPPI Reviewer has a behind the scenes system to automatically cope with Item Lists that require more of the standard 30 seconds timeout period, however this system was until now optimised for the "default list on very large reviews" and also for the architecture that drove EPPI Reviewer until about a year ago. Since then, the architecture changed, and the number of "demanding Lists of Items" never stopped growing.
We have updated the inner adaptation mechanism in the following ways:

• Even when timeouts do happen against the default 30 seconds allocation, the system will now either succeed or fail faster, total allowed time is now 1m and 30s for all lists, except the default (all included items) list, which still has up to 5 minutes to produce results.
• The default list now uses a much better system to cope with time out errors, so should fail extremely rarely (even in huge 1M+items reviews) and generally return results faster than before (for big reviews).
• Other lists have a new, optimised system to react to time out errors, which means they should also be much more reliable now and, concurrently, return results in noticeably less time (for lists that do need more than 30 seconds in the first place).
• The system also optimises the database workload across reviews, making EPPI Reviewer as a whole less affected by spikes of activity generated by our growing user-base.

Enhancement: Full Text indexes

EPPI Reviewer allows to run sophisticated "full text" searches against the following fields: Titles, Abstracts (and both Title and Abstract), Extracted Text from uploaded documents and Text inserted by users in the coding "info box". These searches rely on "full text" indexes which are automatically kept up to date by the SQL database engine. On top of this, we have written a new ad-hoc system to maintain such indexes in a way that maximises their performance, by reorganising or rebuilding them on a weekly basis. This system has been in place for about one month, and has proven to be reliable and effective in speeding up full text searches.

New Feature:

In the "Update review" (Open Alex) pages (available on request), the "filter out" fields for importing "bring up to date" search results were implemented only in EPPI Reviewer Version 4. They are now available in EPPI Reviewer 6 too.

Bug-fix:

Also in the "Update review" (Open Alex) pages, text-based searches against Open Alex would sometimes appear to run normally, but search results could not be imported or browsed. This problem is now solved. We have also updated the user interface slightly, so to better match what the ever-evolving Open Alex API allows.

Bug-fix:

[update June 20] This same release also fixed a bug whereby text-based searches on "uploaded documents" could return the wrong (inflated) number of "hits". When this happened, the search results would claim there were "N" hits, but listing them would return fewer items. Moreover, combining such searches with the AND operator would produce wrong results. This problem is now fixed.

Semi-public new features: GPT4-driven coding on a per item basis

EPPI Reviewer started supporting GPT4-driven coding about six months ago, this is done by using specifically designed coding tools (specifying per-code prompts to submit to GPT4) and on a per item basis. This functionality is available on demand, strictly for evaluation purposes (we know GPT4 makes mistakes, so it would be reckless for us to encourage its use for normal reviewing tasks, at this time). How this system works has changed in the following ways:

• GPT4 now has its own "EPPI Reviewer persona", which means that it will preferentially add coding data using its own "account name". Before, the coding was added "as if" the current user was ticking checkboxes and typing the "info box" text.
• By default, the coding is added as "locked" (there is an option to change this).
• In case human-generated code is already present and "completed", by default the GPT4 robot will add its own coding as "incomplete" (which hides it "behind" the pre-existing completed version). This is the default behaviour as it allows to guarantee that coding done by the machine is always marked as such. However, there is an option which allows to change this behaviour and thus allow GPT4 to operate as it were a human ticking checkboxes normally.

See also this page for more details.

Private new functionality:

This release also includes the possibility of running GPT4 coding jobs against batches of items. This functionality is experimental and was written for two distinct purposes:

1. Allow our own (EPPI Centre) research on the reliability of GPT4 for different coding scenarios.
2. Monitor performance, technical reliability and cost implications, thus opening up the possibility of making this functionality available to all EPPI Reviewer users in a not-so-distant future.

We do not currently know what we'll find through our own research (there are plenty of studies coming out, including a rapid evaluation in which some of us participated), and therefore we do not know if it will ever be possible (or wise) to make GPT4-driven coding a public feature of EPPI Reviewer. However, we think that we might, which is why this preparatory work was necessary.