---
title: "The Loss Function Toolkit"
url: "/learn/series/the-loss-function-toolkit"
---

Every loss function in this corpus is derived, never assumed — and this series
is where that habit gets its full justification, from two directions that turn
out to be the same direction.

## The two routes

**Statistical** — assume a distribution generated the data, then maximise how
likely the data is under it. That is maximum likelihood estimation, and it
produces squared error and cross-entropy as special cases rather than as
defaults.

**Informational** — measure the cost, in bits, of describing data with the
wrong distribution instead of the right one. That is cross-entropy read
through entropy and KL divergence, and it produces the identical loss
functions from an entirely different starting point.

## Why both, not just one

Seeing a loss function derived twice, from unrelated premises, is what makes
"cross-entropy is the natural choice for classification" a fact rather than a
convention worth just memorising.

## Parts

- Maximum Likelihood Estimation: Where Most Loss Functions Actually Come From (unpublished): Maximum likelihood estimation is the method behind nearly every loss function in this corpus, whether or not a model's documentation says so. This post derives the method generally, then shows how assuming Gaussian or Bernoulli noise reproduces squared error and cross-entropy exactly.
- Entropy, Cross-Entropy, and KL Divergence: The Information-Theoretic View of Loss (unpublished): This post builds the information-theoretic account of loss from the ground up — entropy as the average surprise of a distribution, cross-entropy as that same quantity measured against a different assumed distribution, and KL divergence as the information lost when the assumed distribution is wrong.
- Why Cross-Entropy Loss Is the Natural Choice for Classification (unpublished): Two independent routes — maximising likelihood under a Bernoulli or categorical assumption, and minimising the information-theoretic cost of a wrong model — arrive at the identical loss function. This post lays both derivations side by side and explains why that convergence is not a coincidence.