Many machine learning systems involve making predictions, through estimating the value of a dependent variable with an a priori unknown ground truth. Verification of such predictive systems, in particular when applied in high-risk applications, is crucial. Traditional machine learning algorithms and means of validation, however, generally lack capabilities for establishing trustworthy verification; e.g., established common-use validation procedures tend to be biased, leading to error frequency on production data not corresponding with error frequency on test data; and, established common-use validation procedures tend to perform verification on a macroscopic level (per-model) rather than a microscopic level (per-prediction), leading to difficulties in the verification of individual predictions.
The conformal prediction framework offers an alternative method for constructing and evaluating predictive models that appears better suited than traditional predictive methods in applications where thorough verification is crucial. Whereas traditional predictive models output so-called point predictions—a single-valued best-guess prediction for the value of the dependent variable—conformal predictors output multi-valued prediction regions that represent a range of likely value assignments for the dependent variable, constrained by its domain. Any prediction region produced by a conformal predictor comes associated with a very specific, statistically valid, expectation: that the a priori probability of the prediction region containing the ground-truth value of the dependent variable is fixed and known. Under these conditions, model and prediction verification become straight-forward, as each prediction is guaranteed to contain the correct value of the dependent variable with a user-specified probability.
Conformal predictors are able to produce predictions on this form without any knowledge of the shape or parameterization of the data generating distribution (unlike, e.g., Bayesian methods); the only requirement is that the observed data sequence is exchangeable (a looser requirement than the typical i.i.d. assumption). In addition, the conformal prediction framework is model agnostic, in the sense that it can be applied on top of an arbitrary machine learning algorithm (e.g., classification or regression algorithm) and transform the point predictions of the underlying algorithm into prediction regions that exhibit the expected statistical guarantees.
Video Tutorial – Conformal Prediction with Henrik Linusson, Data Scientist, Ekkono Solutions