Implement the cluster sizing formula

CodingBeginner⏱ 30 minAI Systemscapacity planningsizingkv cachebottleneck

Theory: Sizing a Serving Cluster End to End →

Problem

Turn the sizing arithmetic into code that cannot be fudged.

concurrency_ceiling(kv_bytes, per_seq_bytes, efficiency) — how many sequences fit, rounding down.
compute_ceiling(machine_intensity, bytes_per_param) — the batch at which weight-bound decode reaches the machine's compute-to-bandwidth ratio.
replicas_needed(lam, W, rho_target, concurrency) — rounding up.
binding_constraint(memory_ceiling, compute_ceiling_value) — 'memory', 'compute' or 'balanced'.

Then run the worked example: an 80 GB device with 28 GB of weights and 6 GB of workspace, 92% allocator efficiency, 640 MB per sequence, 900 requests per second, 5 s latency, 65% target utilisation.

Implement it, then check the implementation against something you can verify independently.

Write it

Solve it here7 tests · Python runs in your browser

import math

def concurrency_ceiling(kv_bytes, per_seq_bytes, efficiency=1.0):
    """How many sequences a replica can hold at once.

`kv_bytes` is the memory left for KV cache, `per_seq_bytes` the footprint
    of one resident sequence, and `efficiency` the fraction of the budget the
    allocator can actually use. Return the integer ceiling on concurrency —
    round down, since a partial sequence is no use.
    """
    raise NotImplementedError

def compute_ceiling(machine_intensity, bytes_per_param):
    """The batch size at which weight-bound decoding stops being memory-bound.

Weight-limited decode at batch B has arithmetic intensity 2B/b with b
    bytes per parameter. Return the B at which that reaches the machine's
    compute-to-bandwidth ratio, as a float.
    """
    raise NotImplementedError

def replicas_needed(lam, W, rho_target, concurrency):
    """Replicas required to serve lam requests/second at mean latency W while
    holding each replica at utilisation rho_target. Round up."""
    raise NotImplementedError

def binding_constraint(memory_ceiling, compute_ceiling_value):
    """Return 'memory' when the memory ceiling is the smaller of the two,
    'compute' when the compute ceiling is, and 'balanced' when they are equal.
    """
    raise NotImplementedError

⌘/Ctrl + ↵ to run · limit 30s · draft saved in this browser

7 tests will run against your code. The first run downloads the Python runtime — several megabytes, once per browser.

Hints

Hints — 0 of 2 revealed

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
KV cache sizing with MHA and GQA	LLM Ops	Numerical	Beginner	30 min
Size a serving cluster and find the binding constraint	AI Systems	Numerical	Intermediate	1 hour
Build a serving capacity calculator run	LLM Ops	Coding	Advanced	3 hours

Problem

Write it

Hints

Solution

Related problems