What Are Neurons?
An artificial neuron is a weighted sum followed by a non-linearity, which makes it a logistic regression that has stopped claiming to model a probability. This post explains what the non-linearity is actually for, why stacking linear units gains nothing, and how a layer becomes a matrix multiply.
- Level
- intermediate
- Assumes
- Logistic regression and the link function, Matrix multiplication
Not a brain cell
The biological metaphor is a historical accident and it costs more in confusion than it buys in intuition. A neuron is a weighted sum followed by a non-linearity — an equation already met in the previous category.
What the non-linearity is for
Without it, stacking layers gains nothing: a composition of linear maps is a linear map, so a hundred layers collapse into one. The non-linearity is what makes depth mean something.
Choosing one
The choice matters less than its presence, but the gradient behaviour of each option explains a great deal about which architectures trained well and when.
A layer is a matrix multiply
Put many neurons side by side and their weight vectors stack into a matrix. The whole layer becomes one multiplication, which is exactly why this hardware is the hardware.
What it costs
Expressiveness arrives, and the closed-form solution leaves. The loss surface is no longer convex, and fitting becomes a search rather than a calculation.