Subcategory
Clustering and dimensionality reduction — finding structure in data that was never given a label.
There is no loss function handed to you here — no label to be wrong about — which is what makes this subcategory a genuinely different kind of problem rather than a variation on supervised learning.
K-means and hierarchical clustering answer "what groups exist" two different ways, with different assumptions about what a cluster even is. Principal component analysis answers a related but distinct question: which directions in the data carry the information, and which are close enough to noise to drop.
Where k-means demands a number of clusters before it starts, hierarchical clustering builds a full nested structure — a dendrogram — and lets you choose where to cut it afterwards. This post derives agglomerative clustering, the linkage criteria that decide how clusters merge, and when the flexibility is worth the cost.
K-means alternates between assigning points to the nearest centroid and recomputing centroids, and that simplicity hides several sharp edges — sensitivity to initialisation, the assumption of round clusters, and the need to choose the number of clusters in advance. This post derives the algorithm and each failure mode.
Principal component analysis has two equivalent derivations — maximising variance along a direction, and minimising reconstruction error when compressing to fewer dimensions. This post works through the reconstruction framing, connects it back to the singular value decomposition, and shows what a principal component represents.