Version 4.15.4.0 is a release that contains many changes "under the hood". Due to changes in the Microsoft Azure environment, we needed to update a number of our long-standing machine learning services. Specifically, the systems that run the Priority Screening algorithm, as well as those involving all "Binary" classifiers now run in an upgraded Machine Learning platform. As usual, this release also includes enhancements and bugfixes.

Warning: there are important implications for people/reviews which rely heavily on custom-built classifiers, please read the relevant sections of this post carefully and feel free to contact EPPISupport if you wish to discuss these further. We will be offering customised support as default in case the impact of these changes might have a negative impact on your workflow.

General changes

EPPI Reviewer now uses the newest Azure Machine Learning platform for all machine learning tasks, which has a few "general" consequences that apply across the board. First and foremost, the system the EPPI Reviewer uses to control and monitor machine learning jobs is entirely new, and is radically more transparent to developers and support staff. This means that we should now be significantly more efficient in detecting, investigating and/or resolving faults.
The new Azure machine learning environment is a little slower to start jobs going than the older one was (though there are other benefits), so creating new classifiers can take a few minutes longer than before.

Migration of Custom-Built classifiers

Migrating all pre-existing custom-built classifiers was a necessary step, due to changes to the Azure services they rely on. An undesired side effect of this migration is that all custom-built models needed to be rebuilt, though the core algorithms used to build and apply classifiers have not changed; they are exactly the same kind of logistic regression as they were before. On the other hand, the specific implementation of the same core algorithm is subtly different, in that it is now (correctly) less accommodating when fed with too little training data.
Real-world effects, which may have a negative effect (of sorts) are:

1. A small number (about 1%) of custom-built models could not be migrated, as they were built with too little training data (either two or fewer positive or negative examples, or a total sample size of less than 20 items). This shouldn't be a big problem as such tiny models are guaranteed to perform very poorly. These models are now marked as "(failed)" in EPPI Reviewer.
2. All other models have been rebuilt. Rebuilding models requires to set-aside a small, randomly selected subset of records to measure performance. As a consequence, especially for models trained against small datasets (100 items or fewer in the smallest relevant/not relevant class) the real-world performance of the resulting model might have changed. The performance statists for all models have been updated to match the rebuilt models. This however means that the statistics for the models as they were before this release (and thus, presumably, how they were when models got applied) are no longer available in EPPI Reviewer. For this reason we are making this file available, listing all models (identifiable by their IDs) and reporting old and new performance stats.

If you think you might be affected by this, please contact EPPISupport: we can (and will) assist you in re-training models.

Changes to Building Models and Classifying references

Very little has changed on the visible side, although, as stated above, the system that orchestrates building and classifying jobs is entirely new, as is the mechanism in Azure, where jobs are executed. One important change applies to building classifiers: attempting to build/rebuild a classifier will fail with a visible error if/when too little training data has been provided. The new requirements are: both relevant/irrelevant classes need to contain at least 7 records; the training set as a whole needs to contain at least 20 records.

[In EPPI Reviewer 4 all features for building and applying classifiers have been made read-only. Wherever possible, users should use version 6, as Silverlight is now long out of support, and might be completely disabled by a Microsoft update without notice.]

Priority Screening changes

Significant changes were made under the hood, in terms of where/how the training happens, while keeping the training algorithm unchanged. In implementing these changes, we made some marginal improvements to the user interface. Most notable changes are:

1. The training (and prioritising) algorithm may now take between 4 and 12 minutes to run. Before, in small reviews, it could sometimes be faster. Now, the amount of time it will take to run depends mostly on overall activity patterns (determining if the actual execution can start right away) and much less on the amount of data that needs to be processed.
2. Clicking on "(Re)Generate List" used to show the "busy" spinner for at least 15 seconds whenever it did trigger the full re-training procedure. From now on, this delay should be reduced to fractions of a second or 1-2 seconds at most.
3. The "review needs indexing" flag has been removed as it is now redundant. The new prioritising system is taking care of "indexing" at each re-training execution.
4. Because the system is now inherently better at self-reporting its status, we could improve the notification messages shown by the user-interface in response to user actions. These should now appear faster and be more informative.
5. When clicking on the "Refresh" button inside the "Screening\Show progress" panel, the list of training events is now refreshed along with the other "settings" data. Before, this list was being refreshed only upon opening this panel.

[In EPPI Reviewer 4, no changes were made to the User Interface, but all the "under the hood" changes have been implemented here too, so to ensure that priority screening is still available and fully functional in this legacy version too.]

Enhancement to the "Edit coding tools" page/features

In the "Edit coding tools" page, the buttons in the main toolbar now appear darker when the corresponding "activity panel" is open/visible. In this same page, more details about the currently selected node (node type, description) are shown while no "activity panel" is displayed.
In "Edit coding tools", "Item Details" and in "Collaborate\Create New Code" when adding a new code, the code-type picked will be remembered. Thus, when adding another code, the code-type of the last code added will be already selected, whenever possible  .
BugFix: creating a coding tool used to show the new tool as "locked", this was a bug on the user interface (the tool wasn't locked) and is now fixed .

Source code available

Finally, it is our pleasure to announce that we are going to release the source code of EPPI Reviewer and many connected apps during the course of next week. This will aid transparency, especially in terms of our various automation services.