Autoencoders vs PCA

PCA and an autoencoder answer the same request: squeeze each row into fewer numbers, then rebuild the row from those numbers as well as you can. PCA does the squeeze with a straight line or a flat plane, found in closed form. An autoencoder learns the squeeze and the rebuild as two small neural networks. The useful part is knowing where the two agree exactly, and where they part ways.

The idea

Both methods have the same shape. An encoder turns a row $x$ with $d$ numbers into a code $z$ with $k$ numbers, where $k$ is smaller than $d$ . A decoder turns the code back into a row $\hat{x}$ . Training means making $\hat{x}$ as close to $x$ as the code allows. The narrow middle, the bottleneck, forces the method to keep what matters most and let the rest go.

In PCA, both halves are linear. The encoder projects onto $k$ directions and the decoder adds those directions back up. The best directions, in squared error, are the top eigenvectors of the covariance matrix, so there is nothing to train. You get the same answer every time, apart from the sign of each direction.

In an autoencoder, both halves are neural networks. With non-linear activations in the hidden layers, the code can trace a curve or a curved surface, so a single number can describe a point on a bent shape that no straight line fits.

The link worth remembering runs the other way. Take the activations out, so every layer is linear, and train on squared error: the autoencoder ends up spanning exactly the subspace PCA finds. Baldi and Hornik proved in 1989 that, with one hidden layer, this loss has no local minima other than the PCA solution. Stacking more linear layers still computes a single linear map, so it cannot do better. The non-linearity is the whole point.

The maths

Centre the data on its mean $\mu$ and form the covariance matrix $S$ . Its eigenvectors $v_j$ are the principal directions, and each eigenvalue $\lambda_j$ is the variance along its direction.

S = \frac{1}{n} \sum_{i=1}^{n} (x_i - \mu)(x_i - \mu)^{\top}, \qquad S\, v_j = \lambda_j v_j, \quad \lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_d

Dividing by n rather than n − 1 keeps the error identity below exact. It does not change the directions.

Put the top $k$ eigenvectors in the columns of $V_k$ . PCA's reconstruction projects onto them, and its average squared error is the variance it threw away.

\hat{x} = \mu + V_k V_k^{\top} (x - \mu), \qquad \frac{1}{n} \sum_{i=1}^{n} \lVert x_i - \hat{x}_i \rVert^2 = \sum_{j > k} \lambda_j

The Eckart–Young theorem says no rank-k linear map does better in squared error.

An autoencoder replaces the projection with two learned functions, an encoder $f_\theta$ and a decoder $g_\phi$ , and fits them by gradient descent on the same loss.

z = f_\theta(x), \quad \hat{x} = g_\phi(z), \qquad \min_{\theta,\, \phi} \; \frac{1}{n} \sum_{i=1}^{n} \bigl\lVert x_i - g_\phi\bigl(f_\theta(x_i)\bigr) \bigr\rVert^2

If f and g are linear, the best g∘f is PCA's projection. If they bend, the reconstruction can follow a curved surface.

The price of bending is everything PCA never asks for: a choice of layer sizes and activations, a learning rate, a random starting point, a stopping rule, and a fitted shape that can wrap itself around noise. PCA has one decision, $k$ , and the share of variance kept, $\sum_{j \le k} \lambda_j / \sum_j \lambda_j$ , helps you make it.

Try it

The widget squeezes 80 points into one number each and rebuilds them. The dashed line is PCA's answer. The red curve is a small autoencoder, 2 to 6 to 1 to 6 to 2 units, trained right here in your browser with 1,200 steps of Adam. Both are scored on the training points and on 80 fresh points from the same curve.

Curvature of the data1.00

Noise0.05

Autoencoder layers

Show errors for

Seed 1

Training points
PCA, one component
Autoencoder, one-number code

The first principal component keeps 78% of the variance.

Mean squared reconstruction error per point, on the 80 training points and on 80 fresh points from the same curve.
Method	Training	Fresh
PCA	0.0921	0.0911
Autoencoder (tanh)	0.0024	0.0033

On fresh points PCA's error is 0.0911 and the tanh autoencoder's is 0.0033. The autoencoder follows the bend that PCA's line cannot.

Seeded points and a seeded network start, so each setting trains to the same curve on every visit. The red curve is the autoencoder's whole one-number world: every rebuilt point lands on it.

Switch the layers to Linear. The red curve straightens onto the dashed line and both error columns match PCA to four decimal places, as Baldi and Hornik predict.
Back on tanh at curvature 1, the curve follows the bend and its error on fresh points is more than twenty times smaller than PCA's.
Drag curvature to 0. Now the data really is a line, PCA keeps nearly all the variance, and the autoencoder has nothing to add.
Keep curvature at 0, set noise to 0.20 and press New data twice to reach seed 3. The tanh autoencoder beats PCA on the training points and loses to it on fresh ones. It has bent itself around noise.
Show errors for PCA on a curved setting. The error lines are longest at the two ends of the bend, where a straight line strays furthest from the data.

Where I used it

Easy to get wrong

Five traps

Skipping the scaling. PCA chases variance, so a column measured in millimetres outweighs the same quantity in metres. Standardise the columns unless their units are genuinely comparable. Autoencoders train better on scaled inputs too.
Judging on training error alone. A flexible autoencoder can always shave a little more off the training error. Score both methods on held-out rows, as the widget's second column does.
Expecting a deep linear network to beat PCA. Without a non-linear activation, extra layers reach the same subspace, only slower and from a random start.
Reading an autoencoder's code like PCA loadings. Principal components are orthogonal and ordered by variance. Code units are neither, and retraining with a new seed can mix or flip what each unit means.
Reaching for the autoencoder first. If one or two components already keep most of the variance, there is little bend left to capture. Fit PCA first, as the PCA topic describes, and treat it as the baseline to beat.

Sources

Neural networks and principal component analysis: Learning from examples without local minimaBaldi and Hornik, Neural Networks 2(1), 1989, pp. 53–58.The proof behind the first bullet in Try it. A linear autoencoder on squared error has PCA's subspace as its only minimum.
Nonlinear principal component analysis using autoassociative neural networksKramer, AIChE Journal 37(2), 1991, pp. 233–243.The bottleneck design the widget uses: a mapping layer, a narrow code and a de-mapping layer, presented as non-linear PCA.
Reducing the dimensionality of data with neural networksHinton and Salakhutdinov, Science 313(5786), 2006, pp. 504–507.The paper that made deep autoencoders practical, with side-by-side pictures of PCA and autoencoder codes.
Deep Learning, Chapter 14: AutoencodersGoodfellow, Bengio and Courville, MIT Press, 2016.A free textbook chapter covering undercomplete, regularised and denoising autoencoders, with the link to manifold learning.
Principal component analysis: a review and recent developmentsJolliffe and Cadima, Philosophical Transactions of the Royal Society A 374, 2016.A clear modern review of PCA itself, including scaling choices and how many components to keep.