Conformal Prediction

Many machine learning systems involve making predictions, through estimating the value of a dependent variable with an a priori unknown ground truth. Verification of such predictive systems, in particular when applied in high-risk applications, is crucial. Traditional machine learning algorithms and means of validation, however, generally lack capabilities for establishing trustworthy verification; e.g., established common-use validation procedures tend to be biased, leading to error frequency on production data not corresponding with error frequency on test data; and, established common-use validation procedures tend to perform verification on a macroscopic level (per-model) rather than a microscopic level (per-prediction), leading to difficulties in the verification of individual predictions.

The conformal prediction framework offers an alternative method for constructing and evaluating predictive models that appears better suited than traditional predictive methods in applications where thorough verification is crucial. Whereas traditional predictive models output so-called point predictions—a single-valued best-guess prediction for the value of the dependent variable—conformal predictors output multi-valued prediction regions that represent a range of likely value assignments for the dependent variable, constrained by its domain. Any prediction region produced by a conformal predictor comes associated with a very specific, statistically valid, expectation: that the a priori probability of the prediction region containing the ground-truth value of the dependent variable is fixed and known. Under these conditions, model and prediction verification become straight-forward, as each prediction is guaranteed to contain the correct value of the dependent variable with a user-specified probability.

Conformal predictors are able to produce predictions on this form without any knowledge of the shape or parameterization of the data generating distribution (unlike, e.g., Bayesian methods); the only requirement is that the observed data sequence is exchangeable (a looser requirement than the typical i.i.d. assumption). In addition, the conformal prediction framework is model agnostic, in the sense that it can be applied on top of an arbitrary machine learning algorithm (e.g., classification or regression algorithm) and transform the point predictions of the underlying algorithm into prediction regions that exhibit the expected statistical guarantees.

Video Tutorial – Conformal Prediction with Henrik Linusson, Data Scientist, Ekkono Solutions

Related content

More Research from Ekkono

Federated Learning

Federated Learning

Federated Learning is a machine learning technique to train a global model from different clients or devices while keeping the data decentralized.  A central server is in charge of coordinating the transfer of the models’ data between the server and the clients,...

Incremental Learning

Incremental Learning

Incremental Learning is a key feature when dealing with tiny and embedded devices. Many have claimed that only inference can be performed on embedded devices. At Ekkono, we have built a library that can train and run machine learning algorithms not only on big...

Change Detection

Change Detection

On edge machine learning we are constantly dealing with non-stationary environments. These non-stationary environments can range from changes in temperature or humidity in sensors to changes in the way a machine, such as a gas turbine, is operating. Time dependent...