---
title: "Foundations"
url: "/learn/deep-learning/foundations"
category: "deep-learning"
---

Before architectures, before frameworks, before any of the parts that change
every eighteen months: a unit that computes, the non-linearity that makes
stacking those units worthwhile, a method for assigning blame across a stack
of them, and the history of how that stack was arrived at.

Worth working through by hand once. After that you can let a framework do it
and still know what it is doing.

## Posts

- [What Are Neurons?](/learn/deep-learning/foundations/what-are-neurons.md): An artificial neuron is a weighted sum followed by a non-linearity, which makes it a logistic regression that has stopped claiming to model a probability. This post explains what the non-linearity is actually for, why stacking linear units gains nothing, and how a layer becomes a matrix multiply.
- Activation Functions: Why ReLU Won and What Problem It Actually Solves (unpublished): Every activation function solves the same structural problem — making depth meaningful — but they differ enormously in gradient behaviour, and that difference decided which ones survived. This post compares sigmoid, tanh and ReLU on exactly that basis, and explains ReLU's dominance via the vanishing gradient problem.
- From Perceptron to Multilayer Network: A Short History With a Point (unpublished): The single perceptron's inability to represent XOR very nearly ended neural network research, and the resolution — stacking perceptrons into layers — is the entire origin of "deep" in deep learning. This post tells that history because the technical point behind it still explains what depth buys you today.