Gaussian Processes

Most regression models fit a function and hand you a single predicted value — with no honest sense of how much to trust it, especially in regions where you have little data. A Gaussian process (GP) does something more powerful: it returns a prediction and a principled uncertainty band that automatically widens where data is sparse and tightens where it's dense. It's Bayesian regression done not over a fixed equation's parameters, but over entire functions — and that shift is what gives it its uncannily honest uncertainty.

It's a genuine gap worth filling, and it ties together several threads: it's Bayesian, the spatial kriging on that page is a GP, and its uncertainty connects to conformal prediction. This page is the idea — a distribution over functions — the kernel that powers it, how conditioning on data produces the prediction, and where it shines (and doesn't).

A distribution over functions

The conceptual leap that makes GPs special: instead of assuming a form for the function (linear, quadratic) and estimating its parameters, a GP puts a probability distribution directly over the space of all possible functions, then narrows it down using the data. Before seeing data, the GP represents "any smooth function is possible"; after seeing data, it becomes "functions that pass through (or near) these points, and could do anything in between."

This is non-parametric — there's no fixed equation with a fixed number of coefficients; the model's complexity grows with the data. And because it's a distribution over functions, the prediction at any point is itself a distribution — a mean and a variance — which is exactly where the honest uncertainty comes from.

The intuition: jointly Gaussian

The formal definition is surprisingly clean: a Gaussian process is a collection of random variables, any finite subset of which is jointly Gaussian. In plainer terms — for any set of input points, the function values at those points follow a multivariate normal distribution. A GP is fully specified by a mean function (often just zero) and a covariance function:

f(x) \sim \mathcal{GP}\big(m(x),\, k(x, x')\big)

The whole behaviour of the model lives in that covariance function $k(x, x')$ — the kernel — which says how correlated the function's values are at two inputs $x$ and $x'$ . That's the heart of the method, so it's worth dwelling on.

The kernel: where the assumptions live

The kernel encodes your prior beliefs about the function, and it's the one real choice you make. Its core idea is intuitive and familiar: points close together in input space should have similar output values — exactly Tobler's first law from spatial statistics, which is no coincidence, because kriging is a GP.

The most common kernel (the RBF / squared-exponential) makes the correlation between two points decay smoothly with distance, controlled by a length-scale — small length-scale means the function wiggles fast (only very nearby points are correlated); large means it's smooth and slow-varying. Other kernels encode periodicity (for seasonal data) or roughness. Choosing the kernel is how you tell the GP what kind of function to expect — and getting it right is most of the modelling work.

Conditioning on data: the posterior

Here's the magic, and it's pure Bayesian updating. Start with the GP prior (all smooth functions, per the kernel). Observe some data points. Condition the GP on them — and because everything is jointly Gaussian, the maths works out in closed form: the result is another GP, the posterior, with an updated mean and covariance.

A Gaussian process fit. The posterior mean (the line) passes near the observed points; the shaded band is the uncertainty. It pinches tight at the data and balloons wide in the gaps between and beyond them — the model honestly saying 'I don't know out here'.

The posterior mean is your best prediction; the posterior variance is the uncertainty — and the crucial, beautiful property is that the variance shrinks near observed data and grows away from it. The GP knows what it doesn't know: ask it to predict far from any data and it says so, with a wide band, rather than confidently extrapolating nonsense. That calibrated, location-aware uncertainty is what no ordinary regression gives you for free.

Where it shines

GPs earn their keep where uncertainty and small data matter:

Bayesian optimisation — the killer application. To tune expensive things (model hyperparameters, experiment settings) with few evaluations, a GP models the objective and its uncertainty, and you sample next where the GP is both promising and uncertain — efficiently exploring with minimal trials.
Spatial & geostatistics — kriging is exactly a GP over space; predicting a quantity between sample sites with uncertainty is GP regression (a natural fit for climate/environmental work).
Small-data science — when data is scarce and expensive (experiments, simulations), a GP's flexibility and built-in uncertainty beat a big model that would overfit.

The honest limits

GPs are elegant but not universal:

Where it shows up in my work

Honest uncertainty on modest data

GPs are the tool I reach for when data is limited and the uncertainty itself matters — which is common in scientific and risk work (and was directly relevant to the spatial/climate side of my CSIRO work, where kriging — a GP — predicts between measurement sites with honest error bars). The thing GPs give that ordinary models don't is uncertainty that widens where there's no data, so the model openly admits where it's guessing rather than extrapolating with false confidence.

That makes them a natural partner to uncertainty quantification and a sibling of the Kalman filter (both are Gaussian, both track uncertainty). The discipline is knowing the O(n³) ceiling — GPs are for small-to-medium data, not millions of rows — and that the kernel choice carries the assumptions. Used in their sweet spot, they're one of the most elegant ways to be honest about what a model does and doesn't know.

Refresh in 60 seconds

The distribution-over-functions framing, the kernel/covariance role, the closed-form posterior, and the O(n³) scaling limit reflect current Gaussian-process references alongside Bayesian-ML coursework.