---
title: "Architectures"
url: "/learn/deep-learning/architectures"
category: "deep-learning"
---

A plain stack of fully connected layers can, in principle, approximate almost
anything — and is, in practice, a poor way to learn most of it. Architecture is
the set of structural assumptions that make a specific kind of data efficient
to learn from, and this subcategory takes each major assumption in turn:
weight sharing over space, weight sharing over time, and the attention
mechanism that made both look optional.

## Posts

- Attention Is All You Need, Actually Explained (unpublished): Attention replaces the assumption that information must pass step by step through a sequence with a direct, learned weighting over every position at once. This post derives queries, keys and values from the problem attention solves, rather than starting from the equation, and shows why removing recurrence removed the vanishing gradient problem with it.
- Convolutional Neural Networks: Why Weight Sharing Works for Images (unpublished): Convolution is weight sharing applied to spatial data — the same small filter, reused at every position, on the assumption that a pattern useful in one part of an image is useful anywhere in it. This post derives the convolution operation, pooling, and why this bias reduces parameters without reducing what the network can represent.
- Recurrent Networks and the Vanishing Gradient Problem That Killed Them (unpublished): A recurrent network applies the same weights at every step of a sequence, which lets it handle arbitrary length — and also causes its gradients to shrink or explode geometrically over long sequences. This post derives both the architecture and the vanishing gradient problem from the same repeated-multiplication structure.
- The Transformer Architecture, Block by Block (unpublished): Multi-head attention, positional encoding, residual connections and layer normalisation each solve a distinct problem, and the transformer is their combination rather than one idea. This post builds the architecture block by block, motivating each component before assembling the full encoder-decoder structure.