Learning path

The linear algebra, matrix calculus and analysis that every later page assumes without saying so. Most of the difficulty people attribute to machine learning theory is actually unfamiliarity here.

You can thendifferentiate matrix expressions without index gymnastics, and read a spectral argument without stopping.

Measure-theoretic probability at working depth: what a random variable is, what conditioning means, and which mode of convergence a theorem is actually asserting.

You can thentell almost-sure from in-probability convergence, and say what a conditional expectation is without waving at "averaging".

The technical core of modern learning theory: sub-Gaussian tails, concentration inequalities, covering numbers and the geometry of high dimension. Nearly every guarantee later in the path bottoms out here.

You can thenprove a concentration bound from scratch and read "with probability at least 1 − δ" as a quantitative statement rather than a ritual.

What happens to spectra and estimators when the dimension grows with the sample size — the regime every modern model lives in.

You can thenpredict the spectrum of a sample covariance matrix, and say why an estimator that is consistent for fixed d can fail when d ≍ n.

Convexity, smoothness and strong convexity, then complete convergence proofs: gradient descent, projections and proximal steps, duality, acceleration and second-order methods.

You can thenderive a rate rather than quote one, and recognise which assumption a published rate is buying its constant from.

SGD and its descendants: step-size schedules, variance, the difference between convergence of the iterates and of the averages, and what adaptive methods do and do not guarantee.

You can thenstate the assumptions under which SGD converges, and explain precisely why "Adam converges" is not a theorem in the form it is usually quoted.

Empirical risk minimisation, uniform convergence, VC dimension, Rademacher complexity — and where the classical picture stops describing what deep networks do.

You can thenderive a generalisation bound, and say exactly which of its assumptions modern practice violates.

The models whose mathematics is fully understood: linear and ridge regression, logistic regression, kernels and RKHS, SVMs, nearest neighbours, trees and ensembles.

You can thenderive each method from an objective rather than recall its formula, and know which ones have closed-form solutions and why.

Approximation theory, backpropagation as reverse-mode differentiation, convolution and normalisation, attention — followed by an honest account of what is proven about why any of it trains.

You can thenderive backpropagation and attention from first principles, and separate the theorems about deep learning from the folklore.

Autoregressive factorisation and cross-entropy, preference optimisation, scaling laws, and the geometry that makes dense retrieval work — and sometimes fail.

You can thenderive the DPO objective, read a scaling-law fit critically, and compute a retrieval pipeline’s recall ceiling before blaming the generator.

What changes when you stop training from scratch: the low-rank hypothesis behind LoRA, the memory arithmetic that decides what fits on a GPU, and what is actually known about forgetting.

You can thencompute a training run’s memory budget before launching it, and say why LoRA does not reduce activation memory.

The systems half: rounding error and where it goes, the Hessian-weighted objective behind GPTQ, the roofline that separates prefill from decode, and the queueing arithmetic of a server.

You can thenpredict whether a workload is compute- or memory-bound from first principles, and explain why a few outlier channels break per-tensor INT8.

What it takes to put a model in front of real traffic: the queueing theory that decides how hard you can load a server, the arithmetic of tail latency under fan-out, the communication cost of every way of splitting a model across devices, and where the money goes.

You can thensize a serving cluster from a latency target rather than by guessing, and say which of parallelism, caching or batching a given bottleneck actually calls for.

The software around the model. Everything a request passes through before and after the forward pass: the gateway that authenticates and meters it, the limiter that decides whether it runs, the router that picks a GPU, the cache that may answer it outright, the tools it calls, and the pipelines, evaluations and monitors that keep the whole thing honest over time.

You can thendesign and build the service around a model — its API, its limits, its routing, its caches, its data pipeline and its monitoring — rather than only the model call itself.

Multi-step tool-using systems, where per-step reliability is raised to the power of the horizon: compounding error, the bandit view of tool choice, and what inference-time search buys.

You can thencompute an agent’s trajectory success before building it, and price test-time search in both compute and KL divergence.