---
title: "Anatomy of a Modern Neural Network"
url: "/learn/series/anatomy-of-a-modern-neural-network"
---

A diagram of a network says nothing about whether it will train. This series
is about the five decisions that decide that instead, in the order a training
run actually depends on them.

## Why this order

Non-linearity first, because it decides whether depth means anything at all.
Initialisation second, because it decides whether the first forward pass even
produces usable numbers. Normalisation and regularisation follow as the
corrections that keep a working network working past the first few steps. The
optimiser comes last because it assumes everything before it is already sound
— an optimiser cannot fix a network that was never going to train.

## Parts

- Activation Functions: Why ReLU Won and What Problem It Actually Solves (unpublished): Every activation function solves the same structural problem — making depth meaningful — but they differ enormously in gradient behaviour, and that difference decided which ones survived. This post compares sigmoid, tanh and ReLU on exactly that basis, and explains ReLU's dominance via the vanishing gradient problem.
- Weight Initialization: The Silent Variable That Decides If Training Even Starts (unpublished): Initialised too small, activations shrink to nothing layer by layer; initialised too large, they explode — both failures happen before a single gradient update improves anything. This post derives the variance-preserving reasoning behind Xavier and He initialisation, and why the right scale depends on the activation function chosen.
- Batch Normalization and Why Training Got So Much Easier (unpublished): Batch normalisation re-centres and rescales a layer's activations using statistics from the current batch, and that simple operation removed much of the fragility that made deep networks hard to train. This post derives the operation, its effect on the loss landscape, and the difference between training-time and inference-time behaviour.
- Dropout, Weight Decay, and Other Ways to Stop a Network From Memorizing (unpublished): Dropout randomly disables units during training, forcing the network to avoid depending on any one of them too heavily; weight decay penalises large weights directly. This post derives both as regularisation strategies with different mechanisms but the same goal, and compares their effect on what the model actually learns.
- SGD, Momentum, and Adam: A Tour of Optimizers That Actually Matter (unpublished): Stochastic gradient descent's noisy, single-direction updates cause nameable problems — oscillation across narrow valleys, no adaptation to parameters needing different step sizes — and momentum and Adam are direct fixes to those problems, not arbitrary improvements. This post derives each in terms of the failure it corrects.