Subcategory
CNNs, recurrent networks and the transformer — the structural choices that decide what a network can efficiently learn.
A plain stack of fully connected layers can, in principle, approximate almost anything — and is, in practice, a poor way to learn most of it. Architecture is the set of structural assumptions that make a specific kind of data efficient to learn from, and this subcategory takes each major assumption in turn: weight sharing over space, weight sharing over time, and the attention mechanism that made both look optional.
Attention replaces the assumption that information must pass step by step through a sequence with a direct, learned weighting over every position at once. This post derives queries, keys and values from the problem attention solves, rather than starting from the equation, and shows why removing recurrence removed the vanishing gradient problem with it.
Convolution is weight sharing applied to spatial data — the same small filter, reused at every position, on the assumption that a pattern useful in one part of an image is useful anywhere in it. This post derives the convolution operation, pooling, and why this bias reduces parameters without reducing what the network can represent.
A recurrent network applies the same weights at every step of a sequence, which lets it handle arbitrary length — and also causes its gradients to shrink or explode geometrically over long sequences. This post derives both the architecture and the vanishing gradient problem from the same repeated-multiplication structure.
Multi-head attention, positional encoding, residual connections and layer normalisation each solve a distinct problem, and the transformer is their combination rather than one idea. This post builds the architecture block by block, motivating each component before assembling the full encoder-decoder structure.