Series

The Road to Transformers

Three posts tracing why recurrence was the obvious architecture for sequences, why it failed at scale, and what replaced it.

Attention is usually introduced as though it arrived from nowhere. It didn't — it is a direct response to a specific, derivable failure in the architecture it replaced, and this series is written in that order on purpose.

The throughline

Recurrence shares weights across time the same way convolution shares them across space, and that repetition is exactly what causes gradients to vanish over long sequences. Attention removes the repetition by letting every position see every other position directly. The transformer is what you get when that mechanism is built out into a complete architecture.

The arc

Part 1 of 3Recurrent Networks and the Vanishing Gradient Problem That Killed Them

A recurrent network applies the same weights at every step of a sequence, which lets it handle arbitrary length — and also causes its gradients to shrink or explode geometrically over long sequences. This post derives both the architecture and the vanishing gradient problem from the same repeated-multiplication structure.

Part 2 of 3Attention Is All You Need, Actually Explained

Attention replaces the assumption that information must pass step by step through a sequence with a direct, learned weighting over every position at once. This post derives queries, keys and values from the problem attention solves, rather than starting from the equation, and shows why removing recurrence removed the vanishing gradient problem with it.

Part 3 of 3The Transformer Architecture, Block by Block

Multi-head attention, positional encoding, residual connections and layer normalisation each solve a distinct problem, and the transformer is their combination rather than one idea. This post builds the architecture block by block, motivating each component before assembling the full encoder-decoder structure.