Mathematical laboratory
nonconvexcard
The mathematical foundations of modern AI — from probability to optimization, learning theory, deep learning and LLM systems.
Theory with the hypotheses stated, proofs you can fold away until you have tried, seeded simulations you can rerun, and a problem database that goes past the point where the answers are known.
How every topic is built
The same twelve movements, in the same order, on every page. You always know where the proof is, and you always know whether there is one.
- 01IntuitionWhat the result says before any notation.
- 02DefinitionsStated precisely, with the quantifiers in the right order.
- 03FormulationThe object of study, written down.
- 04TheoremsWith every hypothesis spelled out.
- 05ProofsComplete, folded by default so you can try first.
- 06DerivationsStep by step, with the reason for each step.
- 07ExamplesWorked, including ones where the hypotheses fail.
- 08ExperimentA seeded simulation you can rerun.
- 09CodeThe algorithm, implemented.
- 10ProblemsWith hints and full solutions.
- 11PapersWhere this mathematics is actually used.
- 12ResearchWhat is open, stated as open.
Every claim carries its status
Proof libraryThe most common failure in writing about machine learning is presenting an empirical regularity in the grammar of a theorem. Nothing here does that. “SGD converges” is never written; “under assumptions A, B and C, SGD satisfies X” is, with A, B and C stated.
- Theorem
Proved here, or proved in a cited source and marked as such.
- Heuristic
An argument that is useful and is not a proof.
- Empirical observation
A measurement that has been reproduced and not explained.
- Conjecture
Believed, with reasons given, and unproved.
- Open problem
Nobody knows. Stated so you can see the edge.
294 of the 463 results stated on this site are proved in full on their page; the rest are definitions, or are labelled with one of the statuses above.
Fields
All 87 topicsProbability
- Probability3
Probability spaces, expectation, conditional expectation and the four modes of convergence, stated precisely.
- High-Dimensional Probability5
Sub-Gaussian and sub-exponential tails, concentration inequalities, nets, and the geometry of random vectors in large dimension.
- Random Matrix Theory2
Spectral norms of random matrices, singular value bounds, Wishart ensembles and the Marchenko–Pastur law.
- High-Dimensional Statistics2
Estimation when the dimension grows with the sample size: covariance estimation, sparsity, the lasso and minimax rates.
Optimization
- Optimization2
Convex sets and functions, smoothness, strong convexity, and complete convergence proofs for gradient descent.
- Advanced Optimization4
Proximal and projected methods, duality, acceleration, Newton and quasi-Newton methods, with Lyapunov analyses.
- SGD & Optimization Dynamics2
Stochastic approximation, step-size schedules, variance reduction, adaptive methods and their convergence guarantees.
Theory
Models
- Classical Machine Learning6
Linear and logistic regression, nearest neighbours, kernels and RKHS, support vector machines, trees and ensembles.
- Deep Learning Mathematics3
Approximation theory, backpropagation as reverse-mode differentiation, convolutions, recurrence and transformers.
- Modern AI / LLM Mathematics3
Autoregressive factorisation, cross-entropy, attention at scale, preference optimisation and scaling laws.
- Fine-Tuning Mathematics3
Adapting a pretrained model: the low-rank hypothesis behind LoRA, the memory arithmetic that decides what fits on your GPU, and what is actually known about forgetting.
- RAG Mathematics3
Embeddings, similarity geometry, approximate nearest-neighbour search, and retrieval as a latent-variable model.
- Agentic AI Mathematics3
Tool-using agents as decision processes: how per-step error compounds over a trajectory, the bandit view of tool selection, and what inference-time search buys.
Systems
- Quantization Mathematics3
Number formats, rounding error and its propagation, the Hessian-weighted objective behind GPTQ, and why a handful of outlier features breaks naive INT8.
- LLM Ops & Inference Mathematics3
Serving arithmetic: arithmetic intensity and the roofline, KV-cache growth, batching and queueing, and the acceptance algebra of speculative decoding.
- AI System Design15
The mathematics of putting a model into production: Little’s law and the utilisation wall, tail latency under fan-out, parallelism and collective-communication cost models, caching, load balancing, availability and cost per token.
- AI Systems Engineering18
Building the system around the model: gateways, rate limiting and quota, the Model Context Protocol, tool loops, semantic caching, GPU routing, cold starts, data pipelines, drift detection, evaluation harnesses and observability.
A result, as it appears here
Cauchy–Schwarz inequality
The inner product of two vectors is bounded by the product of their norms, with equality exactly for parallel vectors.
Linear Algebra for Machine Learning · Theorem · proof included
From The Abyss
All 38| Solved | Problem | Field | Type | Difficulty | Time |
|---|---|---|---|---|---|
| Derive the optimal weight update abyss | Quantization | Proof | 1 hour | ||
| Exactness of speculative sampling with a modified acceptance rule abyss | LLM Ops | Proof | 1 hour | ||
| Prove Eckart–Young–Mirsky abyss | Fine-Tuning | Proof | 1 hour |
Papers, broken down by the mathematics they use
All 29- Direct Preference Optimization: Your Language Model is Secretly a Reward Model2023
An exact change of variables showing that the KL-regularised RLHF optimum can be expressed directly in terms of the policy, converting the whole pipeline into a single supervised classification loss on preference pairs.
LLMsneeds Research - Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets2022
Documented that on small algorithmic tasks, validation accuracy jumps from chance to near-perfect thousands of optimisation steps after training accuracy reaches 100 percent.
DL Theoryneeds Research - Training Compute-Optimal Large Language Models2022
A refitted scaling law with a corrected methodology, concluding that parameters and tokens should scale in roughly equal proportion — about twenty tokens per parameter — and a demonstration that a four-times-smaller model trained on four-times-more data beats the larger one.
LLMsneeds Research - Training language models to follow instructions with human feedback2022
A three-stage pipeline — supervised fine-tuning on demonstrations, a reward model fit to human preference comparisons, and PPO optimisation of the policy against that reward with a KL penalty — which made a 1.3B model preferred to a 175B base model.
LLMsneeds Research
On sources and honesty
The exposition, examples and problems here are original. Standard references — Vershynin’s High-Dimensional Probability, Boyd and Vandenberghe, Shalev-Shwartz and Ben-David, Wainwright, and the cited papers — are named where a result is standard, and none of them is reproduced. Where a proof is omitted, the page says so and points at a source rather than gesturing. Where nothing is known, the page says that too.
Your progress is stored only in this browser: there is no account, no server and no analytics. The whole site is a static export, so there is nowhere for your data to go.