Subcategory
Entropy, cross-entropy and KL divergence — the information-theoretic account of what a loss function measures.
Maximum likelihood and information theory arrive at the same loss functions from different directions. This category takes the information-theoretic route, because it is the one that explains why cross-entropy is the default for classification rather than merely the convention.
Two independent routes — maximising likelihood under a Bernoulli or categorical assumption, and minimising the information-theoretic cost of a wrong model — arrive at the identical loss function. This post lays both derivations side by side and explains why that convergence is not a coincidence.
This post builds the information-theoretic account of loss from the ground up — entropy as the average surprise of a distribution, cross-entropy as that same quantity measured against a different assumed distribution, and KL divergence as the information lost when the assumed distribution is wrong.