Category
Models that learn from data, derived from first principles: what each one assumes, what it optimises, and where it breaks.
Most machine learning writing starts at the API and works outwards. This starts at the objective function and works towards the API, because the interesting part of a model is what it is trying to minimise and what it had to assume to get there.
The models here are the classical ones, which is the point — they are small enough to derive completely, and everything larger is a variation on them.
What learning means, how gradient descent finds it, and the bias-variance discipline that keeps a model honest.
Every model choice is implicitly a bias-variance tradeoff, and this post makes that explicit in practical rather than purely algebraic terms — what happens as capacity increases, why training and test curves diverge, and how to read that divergence to decide if a model needs to be simpler or more complex.
Gradient descent gets used long before it gets derived, and that gap is where the intuition for tuning it never forms. This post derives it by hand — the update rule, why the learning rate behaves the way it does, and where the method breaks — so it stops being a black box borrowed from a library.
The words "overfitting" and "underfitting" are easy to state and hard to diagnose without seeing the shape of the actual curves. This post walks through training and validation loss curves for both failure modes, and the specific interventions each one calls for.
A model tuned and evaluated on the same data will always look better than it is. This post explains why a validation set must be separate from a test set, what each is actually for, and the specific ways this discipline gets violated in practice — usually without anyone noticing.
Regression, classification and the ensembles built from both — every model here trained on labelled examples.
Unlike every model derived so far in this corpus, a decision tree isn't fit by an optimiser — it's grown by recursively splitting on whichever question reduces impurity the most. This post derives the impurity measures behind that choice and where a greedy tree's structure comes from.
Where logistic regression asks for a probability, a support vector machine asks a purely geometric question — what is the widest possible margin between two classes, and which points define it? This post derives the margin-maximisation objective, support vectors, and a first look at the kernel trick.
Random forests and gradient-boosted trees dominate classical machine learning competitions, and both are built from the same weak base learner combined two structurally different ways. This post derives bagging as variance reduction through averaging, and boosting as sequential error correction.
Clustering and dimensionality reduction — finding structure in data that was never given a label.
Where k-means demands a number of clusters before it starts, hierarchical clustering builds a full nested structure — a dendrogram — and lets you choose where to cut it afterwards. This post derives agglomerative clustering, the linkage criteria that decide how clusters merge, and when the flexibility is worth the cost.
K-means alternates between assigning points to the nearest centroid and recomputing centroids, and that simplicity hides several sharp edges — sensitivity to initialisation, the assumption of round clusters, and the need to choose the number of clusters in advance. This post derives the algorithm and each failure mode.
Principal component analysis has two equivalent derivations — maximising variance along a direction, and minimising reconstruction error when compressing to fewer dimensions. This post works through the reconstruction framing, connects it back to the singular value decomposition, and shows what a principal component represents.
Precision, recall, cross-validation and calibration — the discipline of knowing whether a model is actually good.
Accuracy says nothing about whether a model's predicted probabilities mean anything. Calibration is the property that they do — that among predictions of 80% confidence, roughly 80% are correct — and this post derives how to measure miscalibration and the standard techniques for correcting it.
K-fold cross-validation is simple to implement and surprisingly easy to implement wrong — normalising before splitting, tuning hyperparameters on the folds used to report performance, or splitting time-ordered data randomly. This post derives correct cross-validation and walks through each of these leaks.
A classifier that predicts the majority class every time can have high accuracy and be useless. Precision and recall separate the two ways a model can be wrong, and this post derives both from the confusion matrix, along with the tradeoff between them that every threshold choice makes explicit.