---
title: "Foundations"
url: "/learn/machine-learning/foundations"
category: "machine-learning"
---

Before a specific model, five ideas that apply to all of them: what it means
for a model to have learned anything, the algorithm nearly all of them use to
get there, the tradeoff that explains why more capacity is not free, the
discipline that keeps you from measuring your own bias back to yourself, and
how to recognise when either extreme has gone wrong.

Read these first if the rest of the category is going to mean anything more
than a list of algorithms.

## Posts

- The Bias-Variance Tradeoff: Why More Complex Isn't Always Better (unpublished): Every model choice is implicitly a bias-variance tradeoff, and this post makes that explicit in practical rather than purely algebraic terms — what happens as capacity increases, why training and test curves diverge, and how to read that divergence to decide if a model needs to be simpler or more complex.
- Gradient Descent, By Hand (unpublished): Gradient descent gets used long before it gets derived, and that gap is where the intuition for tuning it never forms. This post derives it by hand — the update rule, why the learning rate behaves the way it does, and where the method breaks — so it stops being a black box borrowed from a library.
- Overfitting and Underfitting, Diagnosed With Real Curves (unpublished): The words "overfitting" and "underfitting" are easy to state and hard to diagnose without seeing the shape of the actual curves. This post walks through training and validation loss curves for both failure modes, and the specific interventions each one calls for.
- Train/Validation/Test: The Discipline That Keeps You From Fooling Yourself (unpublished): A model tuned and evaluated on the same data will always look better than it is. This post explains why a validation set must be separate from a test set, what each is actually for, and the specific ways this discipline gets violated in practice — usually without anyone noticing.