---
title: "From One Model to Many: The Ensemble Story"
url: "/learn/series/the-ensemble-story"
---

A single model has to choose a point on the bias-variance tradeoff. An
ensemble is a way of not fully choosing — and this series builds the argument
for that in four parts.

## The arc

**The tradeoff** — why more capacity in a single model is not free, stated
precisely enough to explain what an ensemble is trying to buy back.

**The weak learner** — a decision tree, cheap enough to fit many of and
unstable enough that combining them helps.

**Two ways to combine** — bagging, which averages away variance, and
boosting, which corrects error sequentially. Structurally different answers to
the same question.

**Honest measurement** — cross-validation, because none of the previous three
parts mean anything if the improvement they claim cannot be measured without
leaking information from the evaluation back into the model.

## Parts

- The Bias-Variance Tradeoff: Why More Complex Isn't Always Better (unpublished): Every model choice is implicitly a bias-variance tradeoff, and this post makes that explicit in practical rather than purely algebraic terms — what happens as capacity increases, why training and test curves diverge, and how to read that divergence to decide if a model needs to be simpler or more complex.
- Decision Trees: How a Model Learns to Ask Good Questions (unpublished): Unlike every model derived so far in this corpus, a decision tree isn't fit by an optimiser — it's grown by recursively splitting on whichever question reduces impurity the most. This post derives the impurity measures behind that choice and where a greedy tree's structure comes from.
- From Trees to Forests: Bagging, Boosting, and Why Ensembles Win Competitions (unpublished): Random forests and gradient-boosted trees dominate classical machine learning competitions, and both are built from the same weak base learner combined two structurally different ways. This post derives bagging as variance reduction through averaging, and boosting as sequential error correction.
- Cross-Validation Done Right (and the Ways It's Usually Done Wrong) (unpublished): K-fold cross-validation is simple to implement and surprisingly easy to implement wrong — normalising before splitting, tuning hyperparameters on the folds used to report performance, or splitting time-ordered data randomly. This post derives correct cross-validation and walks through each of these leaks.