We are currently experiencing a service disruption on our Machine Learning components. Features affected are Priority Screening and Machine Learning classifiers.
These systems use API calls to run and build Machine Learning models - the API run within the Microsoft Azure cloud. Currently most (but not all) API calls are failing, apparently during the low-level stages required to establish an HTTPS connection. Very rarely, the calls succeed, indicating that the problem is likely to be caused by an unexpected fault somewhere - if a misconfiguration was the root cause, we would expect all calls to fail. We are investigating the fault along with Microsoft engineers.
Details on the current disruptions:
For Priority Screening: the vast majority of training events will fail, however, existing priority lists will keep functioning making it possible to "get next item to screen" while the list still contains items to screen. As a consequence, for any given review, priority screening will continue working (without re-training) until the current list has been completed. Each list contains 3000 items, which means that most people might be able to continue working without experiencing any inconvenience (if the fault will be resolved promptly).
For Machine Learning Classifiers: each call to build and apply models is affected, meaning that in most cases it will fail. This will make it impossible to build or apply machine learning models until the fault is resolved.
Although some (very rare) API calls do succeed, we do not recommend trying multiple times, please monitor this thread or our Twitter Feed to receive updates instead.