Category
The mathematics you actually need to read a paper, derive a model by hand, and recognise when a result is wrong.
There is a version of this material that is a maths degree, and a version that is a cheat sheet. Neither is useful. What is useful is the subset that changes how you read a model: enough linear algebra to see what a layer does to a vector, enough probability to know what a loss function is asserting.
Everything here is derived rather than quoted, and every result is followed by the place it shows up in practice.
Vectors, matrices and the decompositions that turn geometry into arithmetic a computer can run.
Every matrix has a handful of special directions it only stretches or shrinks, never rotates — its eigenvectors — and the amount by which it does so — its eigenvalues. This post derives both from the definition, and shows why they describe what a matrix does independent of the basis it was written in.
Backpropagation is matrix calculus applied at scale, and skipping the derivation is why it stays a black box. This post works through differentiating with respect to a matrix, the shapes that fall out of each rule, and the Jacobian bookkeeping a deep learning framework performs on your behalf.
The singular value decomposition factors any matrix, square or not, into a rotation, a scaling and another rotation. This post derives what that factorisation means geometrically, why it always exists where eigendecomposition doesn't, and previews where it resurfaces — PCA, low-rank approximation, and recommender systems.
Derivatives, the chain rule and convexity — the machinery that turns a loss function into something you can actually minimise.
Depth would be useless without an efficient way to compute how the loss depends on an early layer's parameters. This post treats the chain rule not as a calculus exercise but as the load-bearing idea that makes an arbitrarily deep composition of functions differentiable in practice.
Convexity is the property that makes "gradient descent converges to the best answer" a theorem rather than a hope. This post defines it precisely, shows what breaks in non-convex landscapes, and surveys the results explaining why non-convex training still tends to work in practice.
Regularisation, margin maximisation and several classical models are all constrained optimisation problems in disguise. This post derives the method of Lagrange multipliers from geometric first principles, and shows how a constrained problem becomes an unconstrained one with an extra variable.
Random variables, Bayes' rule and maximum likelihood — the reasoning that turns noisy data into a loss function.
Bayes' theorem is a two-line consequence of how conditional probability is defined, but nearly all of its practical value comes from reversing the direction of conditioning — going from how likely evidence is given a cause to how likely a cause is given the evidence.
Behind the familiar "underfitting versus overfitting" picture is an exact algebraic decomposition of expected squared error into three terms — bias, variance and irreducible noise. This post derives that decomposition and shows precisely what choosing "more complex" is trading away, in exchange for what.
Maximum likelihood estimation is the method behind nearly every loss function in this corpus, whether or not a model's documentation says so. This post derives the method generally, then shows how assuming Gaussian or Bernoulli noise reproduces squared error and cross-entropy exactly.
Entropy, cross-entropy and KL divergence — the information-theoretic account of what a loss function measures.
Two independent routes — maximising likelihood under a Bernoulli or categorical assumption, and minimising the information-theoretic cost of a wrong model — arrive at the identical loss function. This post lays both derivations side by side and explains why that convergence is not a coincidence.
This post builds the information-theoretic account of loss from the ground up — entropy as the average surprise of a distribution, cross-entropy as that same quantity measured against a different assumed distribution, and KL divergence as the information lost when the assumed distribution is wrong.