---
title: "Probability & Statistics"
url: "/learn/essential-maths/probability-and-statistics"
category: "essential-maths"
---

Nearly every model in the next two categories is fit by maximising a
likelihood, whether or not its documentation says so. This category makes that
explicit, starting from what a random variable is and ending at the
mathematical statement of why more complex models trade bias for variance.

Read alongside Information Theory if you want the full derivation of where a
loss function's specific shape comes from.

## Posts

- Bayes' Theorem, Properly Motivated (Not Just Stated) (unpublished): Bayes' theorem is a two-line consequence of how conditional probability is defined, but nearly all of its practical value comes from reversing the direction of conditioning — going from how likely evidence is given a cause to how likely a cause is given the evidence.
- Expectation, Variance, and the Bias-Variance Tradeoff's Mathematical Roots (unpublished): Behind the familiar "underfitting versus overfitting" picture is an exact algebraic decomposition of expected squared error into three terms — bias, variance and irreducible noise. This post derives that decomposition and shows precisely what choosing "more complex" is trading away, in exchange for what.
- Maximum Likelihood Estimation: Where Most Loss Functions Actually Come From (unpublished): Maximum likelihood estimation is the method behind nearly every loss function in this corpus, whether or not a model's documentation says so. This post derives the method generally, then shows how assuming Gaussian or Bernoulli noise reproduces squared error and cross-entropy exactly.