Series

From One Model to Many: The Ensemble Story

Four posts on why combining weak models beats building one strong one, from the bias-variance tradeoff through forests to honest evaluation.

A single model has to choose a point on the bias-variance tradeoff. An ensemble is a way of not fully choosing — and this series builds the argument for that in four parts.

The arc

The tradeoff — why more capacity in a single model is not free, stated precisely enough to explain what an ensemble is trying to buy back.

The weak learner — a decision tree, cheap enough to fit many of and unstable enough that combining them helps.

Two ways to combine — bagging, which averages away variance, and boosting, which corrects error sequentially. Structurally different answers to the same question.

Honest measurement — cross-validation, because none of the previous three parts mean anything if the improvement they claim cannot be measured without leaking information from the evaluation back into the model.

The arc

Part 1 of 4The Bias-Variance Tradeoff: Why More Complex Isn't Always Better

Every model choice is implicitly a bias-variance tradeoff, and this post makes that explicit in practical rather than purely algebraic terms — what happens as capacity increases, why training and test curves diverge, and how to read that divergence to decide if a model needs to be simpler or more complex.

Part 2 of 4Decision Trees: How a Model Learns to Ask Good Questions

Unlike every model derived so far in this corpus, a decision tree isn't fit by an optimiser — it's grown by recursively splitting on whichever question reduces impurity the most. This post derives the impurity measures behind that choice and where a greedy tree's structure comes from.

Part 3 of 4From Trees to Forests: Bagging, Boosting, and Why Ensembles Win Competitions

Random forests and gradient-boosted trees dominate classical machine learning competitions, and both are built from the same weak base learner combined two structurally different ways. This post derives bagging as variance reduction through averaging, and boosting as sequential error correction.

Part 4 of 4Cross-Validation Done Right (and the Ways It's Usually Done Wrong)

K-fold cross-validation is simple to implement and surprisingly easy to implement wrong — normalising before splitting, tuning hyperparameters on the folds used to report performance, or splitting time-ordered data randomly. This post derives correct cross-validation and walks through each of these leaks.