Skip to content
nonconvexcard

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov · 2014 · JMLR

Deep LearningLearning Theorydropoutregularisationensembleoverfitting
Read the paper

The problem

Large networks overfit small datasets, and training a true ensemble of many large networks is computationally infeasible.

The contribution

Randomly zeroing units during training, which acts as an implicit ensemble over exponentially many subnetworks with shared weights and is a strong regulariser at negligible cost.

What problem does this paper solve?

Overfitting in overparameterised networks, at a time when explicit regularisers (L2L^2, early stopping) were insufficient and ensembling was too expensive.

What mathematical objects are introduced?

A random mask m∼Bernoulli(1−p)dim⁡m\sim\text{Bernoulli}(1-p)^{\dim} applied elementwise:

a~=11−p m⊙a(inverted dropout, so that Ea~=a).\tilde a = \frac{1}{1-p}\,m\odot a \quad\text{(inverted dropout, so that } \E\tilde a = a).

At test time the mask is dropped and no rescaling is needed.

What assumptions are made?

That the ensemble average is well approximated by the single "mean network" obtained by using all units with scaled weights. This is exact for a linear model and only approximate once nonlinearities are present.

What equations matter?

The exact case is instructive. For a linear model with dropout on the inputs, the expected squared loss is

Em∥y−w⊤(m⊙x)/(1−p)∥2=∥y−w⊤x∥2+p1−p∑jwj2xj2,\E_m\big\|y - w^\top(m\odot x)/(1-p)\big\|^2 = \norm{y - w^\top x}^2 + \frac{p}{1-p}\sum_j w_j^2 x_j^2 ,

so dropout on a linear model is exactly a data-dependent ridge penalty with per-feature weights xj2x_j^2. This identity is the only rigorous characterisation of dropout available, and it does not extend to deep networks.

What theorem or result is important?

No theorem for the deep case. Empirically, consistent gains across vision, speech and text benchmarks, with p=0.5p=0.5 on fully-connected layers becoming the default.

What proof techniques are used?

The linear-model expectation computation above; otherwise experiment.

What prior mathematics is required?

Expectation of a quadratic form under a Bernoulli mask; the bias–variance framing.

Limitations

The ensemble interpretation is heuristic for nonlinear networks. Dropout interacts badly with batch normalisation (the variance shift between train and test compounds), and it is largely absent from modern large transformers, where data scale makes explicit regularisation less necessary.

Why it mattered

It was the default regulariser of the 2012–2018 period and remains standard in smaller models. The linear-model identity is also a useful reminder that a "stochastic trick" can often be resolved into an explicit penalty, which is the right first question to ask of any such method.

What came next