Skip to content
nonconvexcard

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sergey Ioffe, Christian Szegedy · 2015 · ICML

Deep Learningbatch normalisationoptimisationtraining stability
Read the paper

The problem

Deep networks were slow to train and highly sensitive to initialisation and learning rate, apparently because the distribution of each layer's inputs shifts as earlier layers update.

The contribution

Normalising each feature over the mini-batch inside the network, with learned scale and shift, which permits much larger learning rates and reduces sensitivity to initialisation.

What problem does this paper solve?

Training instability in deep networks. The paper's stated diagnosis is internal covariate shift: layer ℓ\ell's input distribution changes whenever layers <ℓ<\ell update, so each layer is chasing a moving target.

What mathematical objects are introduced?

For a feature zz with batch statistics μB,σB2\mu_B,\sigma_B^2,

z^=γ z−μBσB2+ε+β,\hat z = \gamma\,\frac{z-\mu_B}{\sqrt{\sigma_B^2+\eps}}+\beta ,

with γ,β\gamma,\beta learned. At inference, μB,σB2\mu_B,\sigma_B^2 are replaced by running averages, so the train and test computations differ — a subtlety with real consequences.

What assumptions are made?

That batches are large enough and i.i.d. enough for μB,σB\mu_B,\sigma_B to be stable estimates. Both fail for small batches, for sequences of varying length and for distributed training with small per-device batches, which is why transformers use layer norm instead.

What equations matter?

The normalisation above, and the gradient through it, which must account for the dependence of μB,σB\mu_B,\sigma_B on every element of the batch — so the gradient for example ii depends on all other examples in the batch. That coupling is what makes batch norm a genuinely different operation from a per-example rescaling.

What theorem or result is important?

None. The paper reports a 14×14\times reduction in training steps to a given accuracy on ImageNet and demonstrated robustness to larger learning rates.

What proof techniques are used?

None.

What prior mathematics is required?

Backpropagation through a function of batch statistics; the scale-invariance property of the normalised layer.

Limitations

The stated mechanism is not supported. Santurkar et al. injected distribution shift after the batch norm layers and the benefit persisted, which contradicts the covariate-shift account. The train/test discrepancy causes real bugs, and the batch dependence complicates small-batch and distributed training.

Why it mattered

It made much deeper networks trainable and is one of the highest-impact papers in the field. It is also the clearest case study in deep learning of a correct empirical finding with an incorrect explanation — worth remembering when reading any mechanism claim in this literature.

What came next