---
title: "Information Theory"
url: "/learn/essential-maths/information-theory"
category: "essential-maths"
---

Maximum likelihood and information theory arrive at the same loss functions
from different directions. This category takes the information-theoretic
route, because it is the one that explains why cross-entropy is the default
for classification rather than merely the convention.

## Posts

- Why Cross-Entropy Loss Is the Natural Choice for Classification (unpublished): Two independent routes — maximising likelihood under a Bernoulli or categorical assumption, and minimising the information-theoretic cost of a wrong model — arrive at the identical loss function. This post lays both derivations side by side and explains why that convergence is not a coincidence.
- Entropy, Cross-Entropy, and KL Divergence: The Information-Theoretic View of Loss (unpublished): This post builds the information-theoretic account of loss from the ground up — entropy as the average surprise of a distribution, cross-entropy as that same quantity measured against a different assumed distribution, and KL divergence as the information lost when the assumed distribution is wrong.