All topics

Inner products, orthogonal projection, the spectral theorem, the SVD and the matrix norms that every later bound is written in.

Gradients, Jacobians, Hessians and Taylor expansion, set up so that backpropagation and every smoothness assumption later are routine.

Suprema, limits, Lipschitz continuity, compactness and the two or three analysis facts that the probability and optimisation proofs quietly rely on.

The measure-theoretic setup, stated once and precisely, so that later statements about independence, conditioning and convergence have something to mean.

Expectation as an integral, the variance identities used in every bias–variance computation, and conditional expectation as an orthogonal projection.

The four notions of convergence for random variables, which implications hold between them, and the two limit theorems everything downstream depends on.

Orlicz norms, moment generating functions and the equivalent characterisations that make "has a Gaussian-like tail" a usable hypothesis.

Markov to McDiarmid, each with its hypotheses, a complete proof, an interpretation and the regime in which it is the right tool.

Why a Gaussian vector lives on a thin shell, why random directions are nearly orthogonal, and why almost all of a sphere's volume sits near any equator.

How to reduce a supremum over an infinite set to a maximum over a finite one, and what that reduction costs.

A complete proof that random projection to O(log N / eps^2) dimensions preserves all pairwise distances, with the dimension requirement, the geometry and the limits of the result.

The net-plus-concentration argument that gives the sqrt(n) + sqrt(p) bound, and the semicircle law for Wigner matrices.

What the spectrum of an empirical covariance matrix looks like when the dimension is comparable to the sample size, and why that breaks classical statistics.

How many samples are needed to estimate a covariance matrix in operator norm, and what the answer says about the curse of dimensionality.

Restricted isometry, restricted eigenvalues, the lasso error bound, and why sparsity turns an impossible problem into a tractable one.

The four properties that every convergence theorem quantifies over, with the equivalences and the counterexamples that show they are genuinely different.

Four complete convergence proofs — convex, strongly convex, non-smooth and non-convex — with the exact assumptions each one needs.

Weak and strong duality, Slater's condition, and the five KKT conditions — set up so that the SVM derivation is a direct application.

Projection, the proximal operator, mirror descent and Frank–Wolfe, and the single inequality that makes all of their proofs work.

Heavy ball, Nesterov's method, the estimate-sequence proof of the O(1/t^2) rate, and why acceleration is fragile.

Local quadratic convergence, the BFGS update, L-BFGS memory, and why second-order methods are rare in deep learning.

Convex, strongly convex and non-convex SGD; constant versus decaying steps; Robbins–Monro; and the difference between expected and high-probability guarantees.

AdaGrad, RMSProp, Adam and AdamW derived from their design goals, plus SVRG and SAGA and what each one actually guarantees.

Why minimising training error can work, why it needs a uniform bound, and the finite-class and PAC guarantees that follow.

The two standard measures of a hypothesis class's capacity, with complete proofs of Sauer's lemma and the Rademacher generalisation bound.

The exact decomposition, the classical U-curve, the second descent past the interpolation threshold, and what is proved about each.

A careful ledger of what is proved about training deep networks, what each theorem excludes, and which questions are genuinely open.

The normal equations as orthogonal projection, the exact bias and variance of ridge, and why the unbiased estimator is rarely the best one.

The log-likelihood, its convexity, the Hessian as a weighted Gram matrix, and what happens when the data is separable.

Consistency of k-NN, the exact rate under smoothness, and the concentration argument that explains why it fails in high dimension.

Impurity and greedy splitting, the exact variance algebra behind bagging, and boosting derived as functional gradient descent.

Positive definite kernels, the reproducing property, a complete proof of the representer theorem, and kernel ridge regression.

Hard and soft margin, the Lagrangian, every KKT condition interpreted, both duals derived in full, the kernel trick, and the margin generalisation bound.

Universal approximation with its exact hypotheses, why it says nothing about learnability, and backpropagation as reverse-mode differentiation on a computational graph.

Equivariance as the defining property of convolution, the Fourier view, the vanishing-gradient computation for RNNs, and the LSTM gating equations derived from it.

Attention derived from scratch, why the scaling is 1/sqrt(d_k), multi-head as a low-rank sum, positional encodings, residual and normalisation placement, and the exact complexity.

The autoregressive factorisation, maximum likelihood as KL minimisation, what perplexity measures, and where the entropy floor comes from.

The preference model, the KL-regularised RL objective, its closed-form optimum, and the algebra that turns it into a supervised loss.

The compute-optimal allocation computation, LoRA's rank argument, and what is actually known about how in-context learning works.

LoRA's parameterisation, why B starts at zero and the update is scaled by α/r, and the low-rank hypothesis that the method rests on but does not prove.

Sixteen bytes per trainable parameter, the activation bill, and why checkpointing every √L layers is exactly the right thing to do.

Where the quadratic penalty in elastic weight consolidation comes from, why the Fisher information is the right curvature, and what the derivation assumes that deep networks violate.

Cosine, dot product and Euclidean distance and the exact relations between them; why high-dimensional near-orthogonality is what makes retrieval possible; and the maximum-of-N problem that limits it.

The complexity of exact search, why tree methods fail in high dimension, and the mathematics of LSH, IVF, product quantisation and HNSW.

RAG written as marginalisation over a latent document, the exact decomposition of its error, and why retrieval failures propagate rather than average out.

Per-step reliability raised to the power of the horizon — why 95% per step is a failing agent, and what majority voting does and does not repair.

What sampling more candidates actually buys — exponentially decaying failure for linear cost, an unbiased pass@k estimator, and the exact KL price of best-of-n.

An agent choosing among tools is running a bandit — and the logarithmic regret of UCB tells you exactly what exploration is worth.

Affine quantizers, the exact error bound, the additive-noise model that is not a theorem, and the clipping–resolution trade-off.

Rounding each weight to the nearest grid point is not optimal — the layer-wise reconstruction objective has a closed-form correction, and that is what GPTQ computes.

A handful of activation channels with huge magnitudes wrecks per-tensor quantization — and an exact diagonal invariance lets you move the difficulty somewhere it hurts less.

Why prefill is compute-bound and decode is memory-bound, derived from the FLOP and byte counts of a transformer forward pass.

The memory equation that decides how many sequences fit, and why static batching wastes a logarithmically growing fraction of the GPU.

A draft model proposes, the target model verifies — and a rejection-sampling argument makes the output distribution exactly the target's, with no approximation.

The vocabulary the rest of this field is written in — offered load, utilisation, throughput, concurrency and latency percentiles — defined precisely enough to compute with.

The one identity that holds for any queueing system whatsoever — L = λW — proved from an area argument, and the sizing rules that follow from it.

Why latency diverges like 1/(1−ρ) as a server approaches saturation, derived exactly for M/M/1 and M/G/1, and why variance costs as much as load.

Why a request that touches a hundred machines is slow even when every machine is fast, and what hedging buys — derived from the distribution of a maximum.

Three models of what happens when you add machines: a hard ceiling, an escape from it, and the case where throughput actually falls.

Putting Little's law, the KV-cache budget and the utilisation wall together into a single sizing formula, and reading off which constraint actually binds.

What a cache of a given size actually buys under realistic popularity skew, and why prefix caching is governed by a different quantity than object caching.

Why sampling two servers and taking the shorter queue is exponentially better than sampling one, and why least-outstanding-requests beats round robin for variable work.

The arithmetic of nines — why chains multiply, why replicas help less than they look like they should, and the floor that correlated failure puts under any amount of redundancy.

Deriving unit cost from hardware price, throughput and utilisation, and the exact price of a tighter latency target.

The alpha–beta model, and why a ring all-reduce moves a bounded amount of data per node no matter how many nodes there are.

Data, tensor and pipeline parallelism compared by the one quantity that decides between them — bytes communicated per FLOP of useful work.

Why shortest-remaining-first minimises mean latency, what it costs the long requests, and why a bounded queue is a feature rather than a limitation.

Why a scaling loop with a delay must keep spare capacity it never uses, and exactly how long a system stays behind after a load step.

How much traffic a canary needs before it can detect a regression, why stopping when the result looks good inflates the error rate, and what a guardrail suite costs in sensitivity.

Everything a request passes through besides the model: the components, what each one owns, and why the boundaries fall where they do.

Streaming, deadlines, cancellation, idempotency and versioning — the parts of a model API that are hard to change once anyone depends on them.

What belongs in the gateway and what does not, how tenancy is resolved, and why quota and rate limiting are different mechanisms with different failure modes.

Token bucket, sliding window and GCRA, what each actually guarantees, and the two problems unique to limiting an LLM: metering tokens you have not generated yet, and doing it across many instances.

What MCP standardises, the three server primitives and what controls each, the message shapes on the wire, and the security properties a host must enforce because the protocol cannot.

Answering a question you have answered before when it is not phrased the same way — the threshold that decides it, the cost of getting it wrong, and what must never be cached.

Why a GPU replica takes minutes to become useful, what that does to a scaling policy, and the four techniques that shorten it.

Input and output checks around the model, what each can and cannot catch, and why the precision/recall trade is a product decision rather than a threshold.

The pipeline that feeds a model — ingestion, validation, chunking, embedding, indexing — and the lineage records that make "why did it answer that?" a question with an answer.

Timeouts, retries, jitter, circuit breakers, bulkheads and graceful degradation — and the arithmetic showing why an unbudgeted retry policy amplifies an outage.

Tracing a request through prefill, tools and retrieval; accounting for tokens and cost per tenant; and keeping prompts out of your logs.

Index types and what each trades, the recall/latency dial, filtering, freshness and sharding.

Shipping a model change: the registry, shadow traffic, canaries, and why a config change deserves more caution than a code change.

The agent loop as a piece of software: schema design, validation, parallelism, truncation, side effects and the termination conditions that stop it running forever.

Choosing which replica serves a request when the replicas are not interchangeable — prefix affinity, queue-aware placement, LoRA multiplexing and draining.

The software inside a replica: the scheduler loop, continuous batching, paged KV memory, preemption, and what disaggregating prefill from decode buys.

What PSI actually measures, how to monitor an embedding distribution, and how to detect quality regressions when the labels arrive weeks late or never.

Turning "is it good?" into a test suite that runs in CI — golden sets, graders, the failure modes of LLM-as-judge, and why a passing eval on a changed config proves nothing.