Single-stream decode latency floors

NumericalIntermediate⏱ 30 minLLM Opslatencybandwidthlower boundquantization

Problem

Using $t_{\text{step}} \ge Nb/\beta$ with $\beta = 3.35$ TB/s per GPU:

Compute the floor and the implied maximum tokens/second for a 7B model at BF16, INT8 and 4-bit.
Do the same for a 70B model at BF16 across 2 GPUs (assume bandwidth adds and ignore the interconnect).
A product requires 30 tokens/s per user. Which of these configurations can meet it, ignoring all other overheads?

Compute the requested quantities and say what they mean. A number without an interpretation is not an answer.

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
Count the FLOPs of a forward pass	LLM Ops	Numerical	Foundation	15 min
Derive the attention crossover length	LLM Ops	Proof	Beginner	15 min
Ridge point and the batch size needed to saturate	LLM Ops	Numerical	Beginner	15 min
Implement kernel ridge regression run	Classical ML	Coding	Intermediate	1 hour
Price a latency target	AI Systems	Numerical	Intermediate	30 min