Varsion 6.15.2.0 includes new features and bug fixes across a wide range of features. In EPPI Reviewer itself, the headline new functionality is the ability to create "comparisons" between outcome measures extracted independently by two or three reviewers. EPPI Vis gains the ability of picking which data-fields to show or hide for references. A number of important bug fixes is also included in this release.
[Update: 22 April 2024] Please see also this post about the recent performance issues and why we believe they are now resolved.
EPPI Reviewer 6: New Features
Reconciliation reports
In EPPI Reviewer 4, the "Reconcile" window includes the possibility of producing a "reconciliation" report which can be used to show all coding from the separate reviewers, with the exclusion of data about extracted outcomes and PDF coding. This functionality is finally appearing in version 6, enriched with Outcomes data too, which can be included in the report in two different forms, "by reviewer" or "matched", see below for the details.
Matching/comparing outcomes between reviewers
When comparing data extracted independently by two or three reviewers, EPPI Reviewer offered a range of sophisticated systems to help spotting and reconciling differences at the level of "codes applied", but was lacking an equivalent system to deal with Outcome measures. Unlike the main "coding" activity, however, extracting outcome measures has an inherent high number of degrees of freedom, making the task less tractable in straightforward computational terms. It is, in a very crucial sense, alike to duplicate detection for references. Thus, the core component of this new functionality is the "outcome matching" algorithm, which attempts to find "the same outcome measure" as extracted by different reviewers. For this purpose, the system considers two outcomes as a "potential match" if their "outcome type", Effect Size (ES) and Standard Error (SE) values do match (exactly).
Having a list of possible matches, in those cases where there are multiple matching candidates, the algorithm then tries to identify the "best possible" match based on the associated Timepoint and Arms, as well as "outcome", "intervention", and "comparison" codes.
Once Outcomes are either matched or not, they are shown in tables as follows:
- Matching outcomes are grouped together in a table, with one row per reviewer. Within the table, columns are highlighted when values therein differ across reviewers.
- Outcomes that did not match are listed afterwards on a per-person basis and presented in a comparable "table format".
This "match outcomes" functionality is now available in three separate places in EPPI Reviewer 6:
- Within the new reconciliation report mentioned above.
- In the same "reconcile" page, within the "detailed view", it is now possible to choose to show outcomes on a per-reviewer basis (as before), or in the new "matched" form.
- Finally, in the "Item Details" page, "Coding Record" tab, it is now possible to obtain a per-item, per coding tool "matching outcomes" standalone report.
Taken together, these three ways to obtain "outcome matches" should streamline both the detection of disagreements in extracted outcome measures and the reconciliation thereof. We are considering the possibility of adding a "copy this outcome from the coding of ReviewerA to the coding of ReviewerB" functionality, which may also prove useful, but might also encourage hasty decisions, which is why we are still considering it. Please feel free to get in touch in case you have a strong preference about this.
Reconcile page
When a high number of disagreed-upon references (50 or more) is present, this page can take a long time to load, this is because it needs to contact the EPPI Reviewer API to receive the coding from multiple reviewers and multiple items. To help people "see" what's happening, we've added a progress counter, which takes the "loading Item X of N" form.
Matching items to OpenAlex
Both versions previously allowed to trigger multiple concurrent matching jobs for the same review. This isn't useful, as the limiting factor is the OpenAlex API, so it doesn't speed up proceedings, but does actually slow down EPPI Reviewer on the server side, and in the case of EPPI Reviewer 6, it could even make it unresponsive. This problem is now solved, and both versions will avoid triggering multiple concurrent matching jobs for the same review. Moreover version 6 will now automatically refresh (every 30 seconds) the "matched" figures after triggering a new matching job, and will keep refreshing until the user leaves the "matches" tab or the matching job concludes. When visiting the "matches" tab, version 6 will also automatically detect if a matching job is running, and will offer to start the "auto refresh" system if that's the case. Finally, we've made the necessary "behind the scenes" changes required to make sure that whenever multiple matching jobs (across different reviews) are running this will not have significant impacts on the overall performance of EPPI Reviewer 6.
EPPI Visualiser
Starting from this release, the setup area in EPPI Reviewer 6 allows reviewers to pick which fields to hide from a given visualisation, EPPI Vis will now respect these settings and not show them in the in the "item details" page. The fields that cannot be hidden/removed are those available in the items list, thus: ItemId, Title, Short Title, Year, Authors and Journal(/parent title) - which are all fields that do not pose licensing/copyright problems.
Importing references from files
The EPPI Reviewer 6 UI now shows the selected import-filter descriptions, which are useful especially because we've added a third "flavour" of the RIS import filter. This is called "RIS extended". Unlike the two other RIS filters, this new filter imports 'Short title' data, and covers more tags/fields as they appear in the Wikipedia RIS Page (which mentions fields that weren't in the original and authoritative, but no-longer maintained, RIS file specification).
The experimental, OpenAI GPT-powered data-extraction feature
This functionality is still very experimental, and as such it is available "for evaluation" purposes only, and only on a per-review basis, when explicitly requested. In version 6, it is now possible to use Screening tools with it, not only "Standard" tools. Moreover, this function now uses data from title and abstract fields, previously it was sending only the abstract to the GPT API.
Small enhancement to the Account Manager
Many users found the "forgot password" functionality confusing, as it asked for "username OR email address", making people assume that it was asking for BOTH username and email address. They would then attempt to provide both, make a mistake in one or the other, and fail to reset their password. We've updated the corresponding dialog, which now asks for the email (only) by default, while still allowing to use the username instead (by clicking on the apposite link).
Bugfixes
A small number of specific "term values" could break the term highlighting system in EPPI Reviewer 6. This short list included "term", "class" and a number of more convoluted possibilities. This problem has been corrected and it shouldn't now be possible to accidentally pick a term value that will break the highlighting mechanism. At the same time, we've implemented some small, but relatively important improvements to the mechanisms to edit the list of terms to be highlighted.
Outcomes creation: when creating a new outcome in EPPI Reviewer 6, and assigning some (characterisation) codes to it at the same time, after saving the new outcome, the UI "forgot" about these codes, thus not showing the checkboxes as ticked, even if the data was saved correctly to the database. Moreover, if the outcome was then immediately edited and saved, but without re-adding the codes to it, this would overwrite/delete the previously saved characterisation codes. This problem is now resolved.
Editing Outcomes: changing the outcome type in EPPI Reviewer 6 wasn't immediately updating the calculated Effect Size and Standard Error values, and was also failing to immediately update the labels for the outcome type-dependent data fields. Both problems are now resolved.
Configurable answer reports running in "outcomes" mode could include outcomes that would not appear anywhere else in the UI (as a result of having deleted codes or coding tools). These cases are (very) complicated to explain, but will not happen anymore.
Zotero Setup: upon setting up a connection to a Zotero group library, if (contrary to instructions) users produced API keys that provide access to multiple Zotero Libraries, the system would produce a misleading error and fail to conclude the setup phase. This problem is now resolved and it is possible (albeit very much not recommended) to use over-permissive API keys to setup a connection to Zotero.