Version 4.10.0.0 represents an important milestone of the development of EPPI-Reviewer Web. We think that this version is now approaching maturity, as it finally supports double/multiple blind coding exercises in full. In particular, EPPI-Reviewer Web now allows to create and use (coding) Comparisons and to manage disagreements via the "Reconcile" screen.

Features implemented in the Web version.

Create and use Comparisons.

These features are accessible in the "Collaborate" tab.
To conclude a double/multiple (blind) coding or screening exercise (where two or more reviewers assessed the same references independently), EPPI-Reviewer allows to identify and manage disagreements via explicit (pairwise) Comparisons. These allow to:

• List references that have been coded in the same way by 2 reviewers (agreements), and/or list the references that have been coded differently (disagreements).
• Keep a record of the number of agreements and disagreements as present at the time the comparison was created.
• Run "Quick coding" reports, which show the different decisions of 2-3 reviewers and (if any) what is the current agreed (completed) version.
• Complete the agreements in bulk. In addition, EPPI-Reviewer Web now allows to "complete and lock" agreements. The locking mechanism is a (recommended) safety feature which prevents people from changing the agreed coding by mistake.
• Access the reconcile window, where disagreements can be evaluated on a per item basis.

To manage complexity, comparisons in EPPI-Reviewer always compare the coding of two reviewers, however, to help handling multiple coding exercises, each comparison can include two or three reviewers and will then summarise all possible pairwise matches (one or three, respectively).

Reconcile Disagreements.

Most comparisons will identify some disagreements. To deal with these, EPPI-Reviewer Web now allows to use the "Reconcile" screen (each comparison will provide a "Reconcile" button, used for this purpose). This screen will list the "disagreed" references, showing what codes each reviewer (if listed in the comparison) have applied to the references in question. Clicking on each code will show further details: the full path of the code within its coding tool will be represented and the additional text entered in the "Info" box will be shown (if any).
The reconcile screen allows to select what version of the coding shown should be completed (indicating the agreed version) and also to lock the coding, for added safety (recommended).
Note: the "Reconcile" screen is optimised to reconcile disagreements produced while screening. It is entirely possible to use this screen also to reconcile coding differences produced in a double/multiple "data extraction" exercise, but limitations apply. Any "coded text" from PDFs will not be shown, moreover this screen does not help matching "disagreed" answers from within the tree, making such exercises effortful, if the data extraction tool is big and complex.
The features dedicated to complex reconciliations are scheduled to reach EPPI-Reviewer Web with the next release.

Complete/Un-complete and Lock/unlock coding on a per reference, per coding-tool basis.

In the item details screen, the "completion" and "locking" status of the coding that applies to an item is represented, for each coding tool, by icons shown at the root of the tools. A red "no entry" sign signifies that coding is either incomplete or absent; when coding is present and complete a green tick appears instead. A special padlock icon shows when the coding has also been locked, in which case the user cannot tick or un-tick any code from the coding tool in question.
Clicking on any one of these icons will now open a small panel, which allows to change the completed and locking status of the "reference / coding tool" pair on the fly. The same panel also shows who "owns" the coding version that will be affected if changes are applied.

Taken together, these features allow all basic functionalities needed to handle double/multiple coding exercises. In the next release we plan to add the special features designed to better support reconciliations done against complex coding tools.

Bugfixes.

EPPI-Reviewer Web is still officially a Beta version, as we are currently concentrating on implementing features as soon as possible. As a consequence, we expect to gradually discover and fix bugs at a higher than normal rate. Indeed, the list of bug fixes included in this release is not exactly short. We encourage all users of EPPI-Reviewer Web to get in touch with us whenever they experience an unexpected behaviour.

Deleting sources forever: this bug affected both versions of EPPI-Reviewer. When a reference within a source contained one or more arms, attempting to "delete forever" (actually erase) a source, the operation would fail.

Creating a new Coding tool: when creating a new coding tool and accepting the default "type" (Standard) would fail and show an error. To succeed, users had to change the (new coding tool) "Type" value at least once.

Automatically create codes for a (machine learning) classification: with the previous release, the relevant "create" button stopped working.

Multiple rare bugs in Item details (Web version): upon testing the behaviour of EPPI-Reviewer Web in cases where complex coding was present (multiple versions, changing "completion" status, when arms are present and coded individually) we identified a few rare bugs, which only manifested when a number of necessary conditions were all met. Describing the complete details might make this post far too long; however, we can report that most of the bugs involved the user interface only, not the underlying data (making them "annoyances" and not "critical bugs"). Only one bug had the potential of creating real problems: when the same code had been applied to a reference both at the "whole study" and "arm" levels, trying to remove the code from the arm would remove the code to the "whole study" instead. The user interface would not display an error, but would represent what did happen accurately. As a result the problem should have been very visible to users (un-ticking the code, while coding for a given arm, would not make the tick disappear), according to our analysis it is unlikely that this bug did affect any real world user.

[Update: 02/07/2019] A bug affected the "classify" feature in EPPI-Reviewer Web ("Search and Classify" tab). When applying a model, the results would mistakenly ignore one and only one of the references affected.

EPPI-Reviewer 4: JSON coding report.

In EPPI-Reviewer 4 (Silverlight version), Quick coding reports can be obtained in "JSON" format. These are used to instruct third-party machine learning tools and are the mechanism used to generate "Evidence Maps" (https://campbellcollaboration.org/explore/making-an-evidence-and-gap-map-egm.html). When a report contained outcomes and at least one of these outcomes was associated with one or more "Outcome classification" codes, the JSON report would crash and fail.

Small improvement: EPPI-Reviewer 4 (Silverlight).

We have updated the default "Get Silverlight" instructions within EPPI-Reviewer 4. When users try to access EPPI-Reviewer version 4 either with a browser that doesn't support Silverlight, or without having Silverlight installed, they will see a "Get Silverlight" page instead. We've updated the contents of that page so to show better instructions and to provide links to the detailed "how to" guides present in the "Manuals" page.

Future plans.

The following release is expected to include the features present in the "Item details"\"coding record" tab. It will also include major upgrade to the new "Review home" tab, which will receive the features currently presented in the (Main Screen, bottom right) "Review Statistics" tab - this should allow users to perform all tasks connected with double/multiple coding, including troubleshooting and managing complex scenarios. Also planned for the next release is the ability to add and perhaps edit codes without having to leave the "Item Details" screen.