Hierarchical & Mixed-Effects Models

Real-world data is rarely a flat, independent list — it comes in groups. Students nested in schools, patients in hospitals, repeated measurements on the same person, readings from the same sensor, cases within a region. And the moment data is grouped, two observations from the same group are not independent — students in one school share its teaching, readings from one site share its conditions. Hierarchical models (also called multilevel or mixed-effects models) are built for exactly this structure, and they resolve it with one genuinely elegant idea: partial pooling — let every group have its own estimate, but let the groups borrow strength from one another.

It's distinct from the GLM on the statistical-modelling page — that handles the shape of the response; this handles the grouping structure — and it's the same broken-independence lesson as spatial statistics, in a different guise. This page is the problem grouped data poses, the two tempting wrong answers, and the partial-pooling solution that beats both.

When data comes in groups

The core problem is that standard models assume independent observations — but grouped data violates that. Observations within a group are correlated (they share whatever makes the group a group), so treating them as independent overstates how much information you really have, and produces over-confident conclusions — the identical trap to spatial autocorrelation, here driven by group membership rather than geography.

Worse, you often genuinely care about the groups: how do schools differ? which sites run hot? You want estimates per group that are honest about how much data each group actually has. That's where the two naive approaches both fall down.

Two tempting, wrong answers

Faced with grouped data, the instinct is one of two extremes — and both are flawed:

Complete pooling — ignore the groups entirely; throw all the data into one model. This erases real group differences, pretending every school is average. You lose exactly the group-level signal you wanted.
No pooling — fit a completely separate model for each group. This overfits small groups wildly: a school with three students gets an estimate based on three noisy points, treated as if it were as reliable as a school with three thousand. Tiny groups produce absurd, untrustworthy estimates.

One throws away the groups; the other trusts each group blindly regardless of its size. The right answer lives between them — and that "between" is the whole insight.

Partial pooling: the best of both

Partial pooling is the elegant compromise: give each group its own estimate, but pull that estimate toward the overall average by an amount that depends on how much data the group has. A data-rich group's estimate stays close to its own data; a data-poor group's estimate is pulled strongly toward the global mean, borrowing strength from all the other groups.

Partial pooling. No-pooling estimates (top) scatter wildly — small groups land at extreme, unreliable values. Partial pooling (bottom) pulls each group toward the overall mean, and pulls the small/noisy groups much harder than the large/confident ones. Estimates become more reliable across the board.

Done formally, this works because the model treats the group effects as themselves drawn from a shared distribution — a distribution of groups — so estimating one group's effect uses information about how groups vary in general. That's the "borrowing strength" made precise, and it's why partial pooling estimates are, on average, closer to the truth than either extreme.

Shrinkage: the visible signature

The visible effect of partial pooling is shrinkage: extreme group estimates get pulled ("shrunk") toward the overall mean, and the smaller and noisier the group, the harder it's pulled. A group with one wild data point doesn't get to claim a wild estimate — the model rightly says "that's probably noise" and drags it back toward average.

Fixed vs random effects

The vocabulary that confuses everyone, made simple. A fixed effect is a single estimated value for a variable you care about specifically and want to compare directly (the overall effect of, say, a treatment). A random effect is the group-level variation, modelled as deviations drawn from a distribution — used when the groups are a sample from a larger population and you care about the variation across them, not each one individually.

A mixed-effects model simply has both — fixed effects for the population-level relationships you're estimating, and random effects for the group structure. The "random" part is exactly what delivers the partial pooling: by assuming the group effects come from a common distribution, the model shares information across them. (This is also why hierarchical models are naturally Bayesian — that shared distribution is a prior on the group effects.)

Random intercepts & random slopes

Groups can differ in two ways, and the model can capture either or both:

Random intercepts — each group has its own baseline level (some schools just score higher overall), but the effect of a predictor is shared across groups.
Random slopes — the relationship itself varies by group (the effect of study hours on grades is stronger in some schools than others). More flexible, and more data-hungry.

Choosing which to allow is a modelling decision: random slopes capture more, but need enough groups and enough data per group to estimate reliably. As always, the structure you put in should match the structure you believe is in the data — and no more, or it won't converge.

Where it shows up in my work

Grouped data, honestly

A great deal of government data is inherently grouped — figures by region, by unit, by office, or repeated measures over time per entity — and the most important thing hierarchical models give me is the discipline to not reach for the two tempting extremes: lumping everything together (erasing real differences between regions) or analysing each group in isolation (giving a tiny region's three data points the same weight as a large one's thousands).

Partial pooling is the honest middle — small or noisy groups get sensibly shrunk toward the average rather than producing alarming, unreliable estimates, which is exactly what you want before acting on a per-region number. It's the same broken-independence lesson as spatial statistics, the same regularisation intuition, and a natural Bayesian structure — one of the most quietly powerful tools for analysing real, messy, grouped data without fooling yourself.

Refresh in 60 seconds

Grouped/nested data (students in schools, regions, repeated measures) breaks independence — observations within a group are correlated.
Two bad options: complete pooling (ignore groups → erase real differences) and no pooling (separate model per group → overfit small groups).
Partial pooling is the fix: each group gets its own estimate, pulled toward the overall mean by how much data it has — borrowing strength across groups.
Shrinkage is the signature — small/noisy groups pulled hardest toward average. It's regularisation: trade a little bias for much less variance.
Fixed effects (population-level estimates you compare) + random effects (group variation from a shared distribution) = a mixed-effects model. Naturally Bayesian.
Random intercepts (group baselines differ) vs random slopes (the relationship differs by group — more flexible, more data-hungry).

The complete/no/partial-pooling framing, the shrinkage-as-regularisation intuition, and the fixed-vs-random-effects distinction reflect current multilevel-modelling references alongside statistics coursework.