Gradient descent, momentum and SGD
Trajectories and suboptimality curves for GD, Nesterov acceleration, heavy ball and SGD on a quadratic.
SGDgradient descentsgdmomentumnesterov
The mathematics behind this
Gradient Descent and Its Convergence Rates
Four complete convergence proofs — convex, strongly convex, non-smooth and non-convex — with the exact assumptions each one needs.
SGD — Convergence Under Every Standard Assumption
Convex, strongly convex and non-convex SGD; constant versus decaying steps; Robbins–Monro; and the difference between expected and high-probability guarantees.