Active & Semi-Supervised Learning

Standard supervised learning has an expensive appetite: it needs lots of labelled examples, and labelling is slow, costly, and often requires a human expert. Yet unlabelled data is usually abundant and nearly free — mountains of text, images, and records nobody has annotated. Semi-supervised and active learning are two strategies for the same real-world problem: learn a good model when labels are scarce, by making the most of the unlabelled data you have, or by being smart about which few examples are worth the cost of labelling.

It's a genuinely distinct corner of ML — neither fully supervised nor fully unsupervised — and a very practical one wherever labels are the bottleneck. This page is both strategies, the assumptions that make them work, and the failure modes that make them backfire.

The labelling bottleneck

The economics are stark: to train a classifier you need labelled examples, and a human has to create each label — a doctor marking scans, an analyst tagging cases, someone transcribing audio. That's the expensive, rate-limiting step. Meanwhile the unlabelled version of that data piles up for free. The natural question follows: can we get most of the accuracy of a large labelled dataset from just a small one, by exploiting the unlabelled abundance? Both methods here answer yes — in different ways.

Semi-supervised: learning from the unlabelled pile

Semi-supervised learning trains on a small labelled set plus a large unlabelled set together, letting the structure of the unlabelled data sharpen the model. Two common mechanisms:

Self-training / pseudo-labelling — train a model on the labelled data, use it to predict labels for the unlabelled data, keep the most confident of those as pseudo-labels, and retrain on the enlarged set. The model bootstraps itself, teaching itself from its own confident guesses.
Label propagation — build a graph connecting similar points, then let the few known labels spread along the edges to their unlabelled neighbours. Labels flow to nearby points like dye through water.

Both turn cheap unlabelled data into extra (approximate) training signal — but only if a key assumption holds.

When does unlabelled data actually help?

Unlabelled data isn't magic — it helps only when its structure tells you something about the labels. The assumptions that make that true:

Cluster assumption — points in the same dense cluster tend to share a label, so the unlabelled data reveals where the clusters (and thus the decision boundary) are.
Manifold assumption — the data lies on a lower-dimensional surface, and labels vary smoothly along it (the low-dimensional structure idea).
Smoothness assumption — points close together in a high-density region should get the same label.

Active learning: choosing what to label

Active learning attacks the cost from the other side. Instead of labelling data at random, it lets the model choose which examples are most worth labelling — then a human labels just those. Since the budget is small, spending it on the most informative points (rather than random ones) yields a far better model per label.

The active-learning loop. Train on the few labels you have; ask the model which unlabelled point it's most unsure about; a human labels just that one; retrain. The model directs its own learning, spending the scarce labelling budget where it helps most.

How does it pick? The common strategies all target informativeness:

Uncertainty sampling — label the points the model is least sure about (near its decision boundary). Resolving those teaches it the most. (Knowing how unsure ties to uncertainty quantification.)
Query-by-committee — train several models; label the points they disagree on most, since disagreement marks the genuinely ambiguous cases.

Self-supervised: the modern cousin

Worth a mention because it powers today's largest models: self-supervised learning manufactures labels from the unlabelled data itself, via a pretext task — predict a hidden part of the input from the rest (predict the next word; fill in a masked patch of an image). No human labels at all, yet the model learns rich, general representations it can then fine-tune on a small labelled set. This is exactly how LLMs are pretrained (next-token prediction is a self-supervised pretext task), and it's the most powerful way yet found to exploit unlabelled data at scale.

Where it goes wrong

Both approaches have a characteristic failure mode worth respecting:

Where it shows up in my work

When labelled cases are rare

In intelligence and government work, labelled examples are often genuinely scarce — confirmed cases are rare and expensive to establish, while unlabelled records are plentiful. That's exactly the setting these methods are built for. Active learning is the more directly useful one: when an expert's time to label cases is the bottleneck, spending it on the model's most uncertain cases rather than a random sample gets a usable classifier from far fewer labels.

What keeps it honest is the failure modes — pseudo-label confirmation bias (the model amplifying its own errors) and active-learning sampling bias (a labelled set that no longer represents the population). Both mean the results need careful validation on a clean held-out set. It ties to the supervised/unsupervised split, clustering (the assumptions), and the human-in-the-loop discipline that runs through the responsible-AI pages.

Refresh in 60 seconds

The problem: labels are expensive, unlabelled data is cheap. Learn well from few labels.
Semi-supervised: train on a little labelled + lots of unlabelled — pseudo-labelling (self-train on confident guesses) and label propagation (spread labels along a similarity graph).
It only helps if assumptions hold — cluster / manifold / smoothness (the unlabelled data reveals where the boundary shouldn't cut). Otherwise it can mislead.
Active learning: the model chooses what to label — uncertainty sampling (label what it's least sure of), query-by-committee. A human-in-the-loop query loop.
Self-supervised learning makes labels from the data itself (pretext tasks) — how LLMs are pretrained.
Pitfalls: pseudo-label confirmation bias (amplifies errors), active-learning sampling bias (unrepresentative labels). Validate on a clean held-out set.

The semi-supervised assumptions, pseudo-labelling/label-propagation methods, active-learning query strategies, and the confirmation-bias caution reflect current references alongside ML coursework.