Skip to content
nonconvexcard

ImageNet Classification with Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · 2012 · NeurIPS

Deep Learningcnnimagenetreludropoutgpu
Read the paper

The problem

Object recognition at scale had plateaued with hand-designed features; no learned model had come close on a large, realistic image benchmark.

The contribution

A deep convolutional network trained on GPUs with ReLU activations, dropout and data augmentation, halving the ImageNet error rate and starting the deep-learning era in vision.

What problem does this paper solve?

ImageNet classification with 10001000 classes and 1.21.2M training images. Prior systems used hand-engineered descriptors with shallow classifiers and had stalled around 26%26\% top-5 error.

What mathematical objects are introduced?

Nothing new mathematically. The paper's contribution is a combination: eight learned layers of convolution and pooling, the ReLU nonlinearity ρ(t)=max⁡(0,t)\rho(t) = \max(0,t), dropout as a stochastic regulariser, and label-preserving data augmentation.

What assumptions are made?

Translation equivariance of the relevant features, which is what makes convolution the right linear map — and, by the characterisation theorem, the only translation-equivariant linear map. Locality: that features are built from small spatial neighbourhoods.

What equations matter?

ρ(t)=max⁡(0,t)\rho(t) = \max(0,t), with derivative 1{t>0}\one\{t>0\}: unlike the sigmoid, the derivative does not decay for large positive input, so the Jacobian product across layers does not shrink from saturation. The paper measured a six-fold training speed-up from this change alone.

Dropout, at rate pp: multiply activations by independent Bernoulli(1−p)(1-p) masks at training time. The expected activation is preserved by rescaling, and the procedure can be read as training an implicit ensemble of 2#units2^{\#\text{units}} subnetworks with shared weights.

What theorem or result is important?

Top-5 error of 15.3%15.3\% against 26.2%26.2\% for the next best entry — a margin large enough to end the argument about whether learned features could beat designed ones.

What proof techniques are used?

None. The paper is entirely empirical, with careful ablations showing that removing any single layer degrades performance.

What prior mathematics is required?

Convolution and its equivariance property; backpropagation on a computational graph; SGD with momentum and weight decay.

Limitations

No theory for any of the choices. The architecture was found by search and intuition. The local response normalisation layer was later shown to be unnecessary and was replaced by batch normalisation. Depth was limited to eight layers by optimisation difficulty, a barrier that residual connections removed.

Why it mattered

It converted a research position into an engineering fact and redirected the field. The specific technical lessons — ReLU for gradient flow, aggressive augmentation, GPU training — became standard within two years.

What came next