Federated Learning

The standard recipe for machine learning is simple: gather all the data in one place, then train a model on it. But a huge amount of valuable data can't be gathered — it's too sensitive, too regulated, or sits in separate organisations or jurisdictions that aren't allowed to pool it. The data is locked in silos, and the obvious approach is a non-starter. Federated learning (FL) is the clever inversion: instead of bringing the data to the model, bring the model to the data — train across all the silos while the raw data never leaves where it lives.

It's increasingly relevant wherever privacy and data-sharing rules bite, and it pairs naturally with the differential-privacy and MLOps pages. This page is the core idea, the federated averaging that makes it work, the genuine privacy benefit (and its limits), and the hard parts that make it more than just "distributed training."

When data can't be moved

The motivating problem is concrete: you want a model trained on data spread across many places — hospitals, agencies, phones, jurisdictions — but the data can't be centralised. Privacy law forbids it, the data is commercially or legally sensitive, or it simply can't leave the device or organisation that holds it. You're stuck: the model you could build from all the data would be far better than what any single silo can train alone, yet you can't combine the data to build it. Federated learning is the way out of that bind.

Bring the model to the data

The core idea is a loop that keeps the data put. A central server holds the current shared model. Each round: it sends a copy to every participating site; each site trains it briefly on its own local data; each site sends back only the resulting model update (the changed weights) — not the data; the server combines those updates into an improved shared model; and the cycle repeats.

The federated learning loop. The server sends the shared model to each site; each trains locally on data that never leaves; only the model updates come back; the server averages them into a better shared model and sends it out again. The raw data stays home throughout.

The result is a model that has effectively learned from all the data, even though no raw data was ever shared or moved. The information flows as model updates, not records.

Federated averaging

The standard algorithm for combining the updates is federated averaging (FedAvg), and it's pleasingly simple: the server takes the model updates from all sites and computes a weighted average — each site's contribution weighted by how much data it has (a site with more data has more say). That averaged model becomes the new shared model. Despite its simplicity, FedAvg is remarkably effective and is the workhorse of practical federated learning.

The privacy angle — and its limit

The headline benefit is privacy: raw data never leaves its home, which sidesteps the biggest risk of centralisation and helps with regulatory compliance. But there's a crucial honest caveat:

Why it's harder than distributed training

FL isn't just "training on many machines." Its distinctive difficulties:

Non-IID data — the big one. Each site's data is different and unrepresentative of the whole (one hospital's patients differ from another's). When the local datasets are very skewed, averaging their updates can pull in conflicting directions, slowing convergence and biasing the model. Standard ML assumes IID data; FL almost never has it.
Communication cost — sending model updates back and forth every round is expensive, especially across many or bandwidth-limited clients; reducing the rounds matters.
Stragglers & reliability — clients (especially devices) drop out, are slow, or vary wildly in capability; the system must tolerate that.

Devices vs silos

Two settings, with different characters:

Cross-device — millions of small, unreliable clients (phones). The famous example is mobile keyboard next-word prediction, trained across phones without uploading what anyone typed.
Cross-silo — a handful of large, reliable participants (hospitals, banks, agencies), each with substantial sensitive data. Fewer participants, higher stakes, and the setting most relevant to organisations that can't legally pool data.

Where it shows up in my work

Collaborating without sharing the data

The exact problem federated learning solves is a real one in government: different agencies or jurisdictions hold sensitive data that legally or practically can't be combined, yet a model trained across all of it would be far more useful than any one body's slice. FL — train locally, share only model updates — is the mechanism for that collaboration without the data-sharing that privacy law and trust would otherwise forbid. It's the cross-silo setting that fits.

What keeps it honest is the privacy caveat: FL is a real improvement but not a guarantee on its own — updates can leak, so it has to be paired with differential privacy and secure aggregation for genuine protection. And the non-IID reality (each agency's data is different) is the practical hurdle that makes it more than distributed training. It completes the privacy-preserving toolkit alongside DP and governance — the set of techniques for getting value from data about people without exposing them.

Refresh in 60 seconds

Federated learning trains a shared model without centralising the data — for when data is too sensitive/siloed to pool. Bring the model to the data.
The loop: server sends the model to each site → each trains locally → sends back only model updates (not data) → server averages → repeat.
Federated averaging (FedAvg): weighted average of the updates (by data size). Simple, effective, the workhorse.
Privacy benefit: raw data stays home — but updates can still leak, so combine with differential privacy + secure aggregation for real protection.
Hard parts: non-IID data (each site's data differs → slow/biased convergence), communication cost, stragglers.
Cross-device (phones — keyboard prediction) vs cross-silo (organisations/agencies — the data-can't-be-pooled case).

The model-to-the-data idea, FedAvg, the update-leakage caveat (DP + secure aggregation), and the non-IID challenge reflect current federated-learning references alongside privacy-ML work.