---
title: "Model Evaluation & Validation"
url: "/learn/machine-learning/model-evaluation"
category: "machine-learning"
---

A model that fits is not the same as a model that works, and this subcategory
is entirely about the gap between the two. It assumes the models from
Supervised and Unsupervised Learning and asks the question neither of those
subcategories can answer on its own: is this any good, and how do you know
without fooling yourself?

## Posts

- Calibration: When Your Model's Confidence Doesn't Match Reality (unpublished): Accuracy says nothing about whether a model's predicted probabilities mean anything. Calibration is the property that they do — that among predictions of 80% confidence, roughly 80% are correct — and this post derives how to measure miscalibration and the standard techniques for correcting it.
- Cross-Validation Done Right (and the Ways It's Usually Done Wrong) (unpublished): K-fold cross-validation is simple to implement and surprisingly easy to implement wrong — normalising before splitting, tuning hyperparameters on the folds used to report performance, or splitting time-ordered data randomly. This post derives correct cross-validation and walks through each of these leaks.
- Precision, Recall, and the Cost of Being Wrong in Different Directions (unpublished): A classifier that predicts the majority class every time can have high accuracy and be useless. Precision and recall separate the two ways a model can be wrong, and this post derives both from the confusion matrix, along with the tradeoff between them that every threshold choice makes explicit.