Machine-Learning

The internet uses a protocol - it’s called http. A protocol, in this sense, is an agreement. It allows any party, wishing to partake in an exchange, to have a clear expectation of how to do things. For the internet, it tells interested parties how to communicate, to send and receive data. It has occurred to me that machine learning currently lacks a protocol. Instead, we have frameworks, which implement protocols defined by the people that have implemented the frameworks. And while there is nothing wrong with these “framework protocols”, the fact that there are several of them is a problem. It implies that there are number of ways to achieve the same thing - read data, transform it, train a model, export it, load it, and get predictions out of it. ...

In this article, I share notes on handling concept drift for machine learning models. Introduction Concept drift occurs in an online supervised learning setting, when the relationship between the input data X and output data y is altered to the extent that a model mapping X to y can no longer do so with the same efficacy. In online supervised learning, there are three types of drift that can occur: (1) feature drift, i.e. distribution of X, (2) real concept drift i.e. relation between X and y or p(y|X), and (3) change in the prior distribution p(y), e.g. new classes arrived. While both feature and prior distribution changes may be interesting to monitor, for purposes that extend beyond understanding changes in the problem space, it is only real concept drift that we are chiefly concerned with. Consider the following scenario: Google decides to increase the price of their flagship Android devices by 20%, making them more appealing to certain segments and less to others whom will detract to lower end versions of the brand or switch brands altogether to acquire devices within the original price range. As a result, the distribution of users signing up from specific mobile platforms may change. This would be a feature drift. If, however, this change is not sufficient to cause the model to err in its ability to, for example, predict user retention or quality of experience because despite the shifts in demographics most users will remain within the same device price range, and therefore have a similar initial experience, there may very well be no real concept drift. ...