---
title: "The Road to Transformers"
url: "/learn/series/the-road-to-transformers"
---

Attention is usually introduced as though it arrived from nowhere. It didn't —
it is a direct response to a specific, derivable failure in the architecture
it replaced, and this series is written in that order on purpose.

## The throughline

Recurrence shares weights across time the same way convolution shares them
across space, and that repetition is exactly what causes gradients to vanish
over long sequences. Attention removes the repetition by letting every
position see every other position directly. The transformer is what you get
when that mechanism is built out into a complete architecture.

## Parts

- Recurrent Networks and the Vanishing Gradient Problem That Killed Them (unpublished): A recurrent network applies the same weights at every step of a sequence, which lets it handle arbitrary length — and also causes its gradients to shrink or explode geometrically over long sequences. This post derives both the architecture and the vanishing gradient problem from the same repeated-multiplication structure.
- Attention Is All You Need, Actually Explained (unpublished): Attention replaces the assumption that information must pass step by step through a sequence with a direct, learned weighting over every position at once. This post derives queries, keys and values from the problem attention solves, rather than starting from the equation, and shows why removing recurrence removed the vanishing gradient problem with it.
- The Transformer Architecture, Block by Block (unpublished): Multi-head attention, positional encoding, residual connections and layer normalisation each solve a distinct problem, and the transformer is their combination rather than one idea. This post builds the architecture block by block, motivating each component before assembling the full encoder-decoder structure.