Sizing best-of-n
NumericalBeginner⏱ 15 minAgentsbest of nsamplingcost
Problem
A model solves a task with probability per attempt, and a checker identifies correct solutions perfectly.
- Compute samples needed for 90%, 99% and 99.9% success.
- Compute the marginal cost of each additional nine.
- If each sample costs $0.002, compute cost per solved task at each target.
Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.
Solution
Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.
Related problems
All problems| Solved | Problem | Field | Type | Difficulty | Time |
|---|---|---|---|---|---|
| Price a latency target | AI Systems | Numerical | 30 min | ||
| The pass@k estimator | Agents | Proof | 30 min | ||
| Temperature monotonically controls entropy | LLMs | Proof | 30 min | ||
| The ceiling imposed by a fallible verifier | Agents | Proof | 30 min | ||
| The KL divergence of best-of-n abyss | Agents | Proof | 1 hour |