Skip to content
nonconvexcard

Deep Residual Learning for Image Recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun · 2016 · CVPR

Deep Learningresnetresidual connectiondepthidentity mappingoptimisation
Read the paper

The problem

Adding layers to a deep network made both training and test error worse, which cannot be an overfitting problem and therefore had to be an optimisation problem.

The contribution

Reparameterising each block to learn a residual F(x) with output x + F(x), which makes the identity map trivially representable and lets networks of hundreds of layers train.

What problem does this paper solve?

The degradation problem. A 5656-layer plain network had higher training error than a 2020-layer one on CIFAR-10. Since the deeper network's function class strictly contains the shallower one's — set the extra layers to the identity — this is not a capacity or generalisation failure. It is a failure to find a solution that provably exists.

What mathematical objects are introduced?

The residual block

y=x+F(x;θ),y = x + F(x;\theta),

where FF is two or three convolution–normalisation–ReLU layers. The reparameterisation is exact: the function class is unchanged, only the coordinates on it are different.

What assumptions are made?

Implicitly, that the target function is closer to the identity than to zero in the relevant sense — that each block needs to make a small adjustment. This is a hypothesis about the problem, and it is why the reparameterisation helps rather than being a cosmetic rewrite.

What equations matter?

The Jacobian of a residual block is

∂y∂x=I+∂F∂x,\frac{\partial y}{\partial x} = I + \frac{\partial F}{\partial x},

so a stack of LL blocks has Jacobian ∏ℓ(I+∂Fℓ/∂x)\prod_\ell\big(I + \partial F_\ell/\partial x\big). When the ∂Fℓ\partial F_\ell are small this product stays near II, whereas the corresponding plain-network product ∏ℓJℓ\prod_\ell J_\ell is generically exponential in LL. This is the mathematical content of the paper.

What theorem or result is important?

No theorem. Empirically: 152152-layer networks train successfully and win ImageNet 2015; the degradation curve is removed; and a 12021202-layer network trains (though it overfits). The accompanying observation that the learned residuals have small norm supports the assumption above.

What proof techniques are used?

None in the original paper. Later work (the identity-mappings follow-up) analysed the forward and backward signal propagation through the skip path more carefully, and the general phenomenon connects to the landscape literature.

What prior mathematics is required?

The chain rule as a Jacobian product, operator norms, and the observation that identity-plus-small matrices compose stably.

Limitations

The explanation is a plausibility argument, not a theorem: there is no result stating that residual parameterisation makes the optimisation problem easier in a precise sense. Very deep ResNets behave in some respects like ensembles of shallower paths, which complicates the "depth is doing the work" reading.

Why it mattered

The additive skip connection is now in essentially every deep architecture, including every transformer, where the residual stream is the central object. The paper also established the diagnostic discipline of distinguishing optimisation failure from generalisation failure by looking at training error.

What came next