Research papers
29 papers spanning 2012–2023, each broken down the same way: the problem it attacks, what it actually contributes, the mathematics it presumes, and what it did not settle. Entries link to the topics that prove the results the paper takes for granted.
These pages describe and cite the papers; they do not reproduce them. Every entry links to the authoritative version — arXiv, a DOI or the authors' own page — and where a paper's central claim is empirical rather than proved, the breakdown says so.
Mathematics used
Decade
Sort by
29 of 29 papers
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn · NeurIPS
ContributionAn exact change of variables showing that the KL-regularised RLHF optimum can be expressed directly in terms of the policy, converting the whole pipeline into a single supervised classification loss on preference pairs.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, Vedant Misra · ICLR Workshop
ContributionDocumented that on small algorithmic tasks, validation accuracy jumps from chance to near-perfect thousands of optimisation steps after training accuracy reaches 100 percent.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark and others · NeurIPS
ContributionA refitted scaling law with a corrected methodology, concluding that parameters and tokens should scale in roughly equal proportion — about twenty tokens per parameter — and a demonstration that a four-times-smaller model trained on four-times-more data beats the larger one.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray and others · NeurIPS
ContributionA three-stage pipeline — supervised fine-tuning on demonstrations, a reward model fit to human preference comparisons, and PPO optimisation of the policy against that reward with a KL penalty — which made a 1.3B model preferred to a 175B base model.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen · ICLR 2022
ContributionFreezing the pretrained weights and training a low-rank additive update, reducing trainable parameters by orders of magnitude with no inference-time cost after the update is merged.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever · ICLR
ContributionSystematic demonstration of a second descent in test error past the interpolation threshold, across model size, dataset size and training time, unified by an effective-model-complexity variable.
Jonathan Ho, Ajay Jain, Pieter Abbeel · NeurIPS
ContributionA model that learns to reverse a fixed Gaussian noising process, whose variational bound simplifies to a plain noise-prediction regression loss — stable to train, with sample quality matching or exceeding GANs.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih · EMNLP
ContributionA dual-encoder trained with a contrastive loss using in-batch negatives and hard negatives, which outperforms BM25 substantially and became the standard dense retriever recipe.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell and others · NeurIPS
ContributionA 175B-parameter autoregressive model that performs many tasks from a prompt containing a few examples, with no parameter updates, and whose few-shot performance improves smoothly with scale.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela · NeurIPS
ContributionTreating the retrieved document as a latent variable and marginalising over it, training the generator and the query encoder end to end through the retrieval distribution.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei
ContributionEmpirical power laws relating test loss to parameters, data and compute over seven orders of magnitude, with the remarkable finding that architectural details matter far less than scale.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova · NAACL
ContributionMasked language modelling, which lets every token attend to both directions during pretraining, plus the pretrain-then-fine-tune recipe that became standard for understanding tasks.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal · PNAS
ContributionIdentified and named the double-descent risk curve, and demonstrated it in random-feature models, random forests and small neural networks, arguing that the interpolation regime is a distinct statistical regime rather than an aberration.
Jonathan Frankle, Michael Carbin · ICLR
ContributionThe hypothesis that a dense network contains a sparse subnetwork which, trained from the original initialisation, matches the full network's accuracy, together with an iterative-magnitude-pruning procedure that finds such subnetworks.
Arthur Jacot, Franck Gabriel, Clément Hongler · NeurIPS
ContributionIn the infinite-width limit with a particular scaling, the tangent kernel becomes deterministic and constant in time, so gradient flow on the network is exactly kernel gradient flow — a linear, solvable system.
Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin · NeurIPS
ContributionAn architecture built only from attention and position-wise feedforward layers, removing recurrence entirely and making the computation over a sequence fully parallel.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton and others · Nature
ContributionA single network trained purely by self-play from random initialisation, with search-derived targets, which exceeded the previous system's strength and generalised to chess and shogi unchanged.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean · ICLR
ContributionA layer with thousands of experts and a learned gate routing each token to a few of them, decoupling parameter count from per-token compute, with a load-balancing loss to prevent expert collapse.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun · CVPR
ContributionReparameterising each block to learn a residual F(x) with output x + F(x), which makes the identity map trivially representable and lets networks of hundreds of layers train.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot and others · Nature
ContributionCombining learned policy and value networks with Monte Carlo tree search, so that the networks prune the search and the search improves on the networks.
Tianqi Chen, Carlos Guestrin · KDD
ContributionA second-order expansion of the loss giving closed-form leaf values and split gains, plus explicit complexity regularisation and systems work for sparsity and out-of-core training.
Diederik P. Kingma, Jimmy Ba · ICLR
ContributionExponential moving averages of the gradient and of its square, with explicit bias correction, giving a per-coordinate step that is invariant to gradient rescaling and requires little tuning.
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift2015Deep Learning
Sergey Ioffe, Christian Szegedy · ICML
ContributionNormalising each feature over the mini-batch inside the network, with learned scale and shift, which permits much larger learning rates and reduces sensitivity to initialisation.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski and others · Nature
ContributionTwo stabilising mechanisms — experience replay and a periodically-frozen target network — that made deep Q-learning work, achieving human-level play on Atari games from raw pixels.
Diederik P. Kingma, Max Welling · ICLR
ContributionThe reparameterisation trick, which moves the sampling outside the parameterised distribution so that the ELBO's gradient can be estimated by ordinary backpropagation with low variance.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov · JMLR
ContributionRandomly zeroing units during training, which acts as an implicit ensemble over exponentially many subnetworks with shared weights and is a strong regulariser at negligible cost.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio · NeurIPS
ContributionA minimax game between a generator and a discriminator whose equilibrium minimises the Jensen–Shannon divergence to the data distribution, requiring no tractable likelihood at all.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean · NeurIPS
ContributionThe skip-gram model with negative sampling, which learns dense word vectors from co-occurrence at a cost linear in the corpus, and the observation that the resulting geometry encodes analogies.
Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · NeurIPS
ContributionA deep convolutional network trained on GPUs with ReLU activations, dropout and data augmentation, halving the ImageNet error rate and starting the deep-learning era in vision.