Series

Anatomy of a Modern Neural Network

Five posts on what turns an architecture into a network that trains — activations, initialisation, normalisation, regularisation, and the optimiser.

A diagram of a network says nothing about whether it will train. This series is about the five decisions that decide that instead, in the order a training run actually depends on them.

Why this order

Non-linearity first, because it decides whether depth means anything at all. Initialisation second, because it decides whether the first forward pass even produces usable numbers. Normalisation and regularisation follow as the corrections that keep a working network working past the first few steps. The optimiser comes last because it assumes everything before it is already sound — an optimiser cannot fix a network that was never going to train.

The arc

Part 1 of 5Activation Functions: Why ReLU Won and What Problem It Actually Solves

Every activation function solves the same structural problem — making depth meaningful — but they differ enormously in gradient behaviour, and that difference decided which ones survived. This post compares sigmoid, tanh and ReLU on exactly that basis, and explains ReLU's dominance via the vanishing gradient problem.

Part 2 of 5Weight Initialization: The Silent Variable That Decides If Training Even Starts

Initialised too small, activations shrink to nothing layer by layer; initialised too large, they explode — both failures happen before a single gradient update improves anything. This post derives the variance-preserving reasoning behind Xavier and He initialisation, and why the right scale depends on the activation function chosen.

Part 3 of 5Batch Normalization and Why Training Got So Much Easier

Batch normalisation re-centres and rescales a layer's activations using statistics from the current batch, and that simple operation removed much of the fragility that made deep networks hard to train. This post derives the operation, its effect on the loss landscape, and the difference between training-time and inference-time behaviour.

Part 4 of 5Dropout, Weight Decay, and Other Ways to Stop a Network From Memorizing

Dropout randomly disables units during training, forcing the network to avoid depending on any one of them too heavily; weight decay penalises large weights directly. This post derives both as regularisation strategies with different mechanisms but the same goal, and compares their effect on what the model actually learns.

Part 5 of 5SGD, Momentum, and Adam: A Tour of Optimizers That Actually Matter

Stochastic gradient descent's noisy, single-direction updates cause nameable problems — oscillation across narrow valleys, no adaptation to parameters needing different step sizes — and momentum and Adam are direct fixes to those problems, not arbitrary improvements. This post derives each in terms of the failure it corrects.