Skip to content
nonconvexcard

Choosing the draft length

NumericalAdvanced⏱ 30 minLLM Opsspeculative decodingoptimisationspeedup

Problem

With acceptance rate α\alpha and relative draft cost cc, speedup is S(γ)=(1−αγ+1)/((1−α)(γc+1))\mathcal S(\gamma) = (1-\alpha^{\gamma+1})/\big((1-\alpha)(\gamma c+1)\big).

  1. For α=0.7\alpha = 0.7, c=0.15c = 0.15, tabulate S\mathcal S for γ=1,…,8\gamma = 1,\dots,8 and find the optimum.
  2. Show that as c→0c \to 0 the optimal γ\gamma diverges, and give the limiting speedup for fixed γ→∞\gamma \to\infty.
  3. A team reports α=0.95\alpha = 0.95 and claims a 10×10\times speedup. Is that consistent?

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Related problems

All problems
SolvedProblemFieldTypeDifficultyTime
Optimal checkpointing frequencyFine-TuningProofIntermediate30 min
Exactness of speculative sampling with a modified acceptance rule abyssLLM OpsProofAdvanced1 hour
Implement gradient descent and verify all four rates runOptimizationCodingAdvanced3 hours
Optimal clipping for a Laplace-distributed tensorQuantizationProofAdvanced1 hour