Skip to content
nonconvexcard

API Design for Model Endpoints

Streaming, deadlines, cancellation, idempotency and versioning — the parts of a model API that are hard to change once anyone depends on them.

AI EngineeringCore⏱ 3 hours6 min readapi designstreamingSSEidempotencydeadlines
AssumesThe Anatomy of an AI System

Streaming

DefinitionThe streaming response contract

A streamed generation is a sequence of typed events over one connection:

event: message_start        { "id": "...", "model": "...", "usage": { "input_tokens": 412 } }
event: content_delta        { "index": 0, "text": "The" }
event: content_delta        { "index": 0, "text": " utilisation" }
        ...
event: message_stop         { "stop_reason": "end_turn",
                              "usage": { "input_tokens": 412, "output_tokens": 218 } }

The contract has three parts:

  1. Exactly one terminal event. A stream that ends without message_stop was truncated. Without this, a dropped connection and a finished answer are indistinguishable to the client.
  2. A stop reason. end_turn, max_tokens, stop_sequence, tool_use, refusal — the client frequently needs to branch on why generation ended, and inferring it from the text is guesswork.
  3. Usage on the terminal event, not only at the start: output tokens are unknown until the end, and this is what the client is billed on.

Key idea

Streaming exists for one reason: time to first token is a different product from time to last token. A 5-second generation that starts emitting at 300 ms feels interactive; the same generation delivered atomically at 5 s does not. That makes TTFT a first-class SLO — see What a Serving System Must Guarantee — and it means the API must not buffer for convenience anywhere on the path.

Deadlines rather than timeouts

HeuristicPropagate deadlines, do not re-specify timeouts

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

The client states when it will give up, as an absolute deadline; every hop passes the remaining budget downstream and refuses work it cannot finish in time.

# each hop subtracts what it has already spent
remaining = deadline - now()
if remaining <= MIN_USEFUL:
    raise DeadlineExceeded            # do not start work that cannot finish
result = await downstream.call(req, deadline=deadline)   # absolute, not relative

Per-hop fixed timeouts have the opposite property: they compose additively, so a four-hop path with 30-second timeouts can take two minutes, and hops keep working on requests whose callers left long ago.

Assumptions

Deadline propagation assumes roughly synchronised clocks, which is why deadlines travel as a remaining duration in practice rather than an absolute timestamp, converted at each hop. It also assumes every hop actually checks; one hop that ignores the budget reintroduces the unbounded case for the whole path.

Retries that do not double-charge

DefinitionIdempotency for non-idempotent generation

Generation is not idempotent: the same request run twice produces two answers and two bills. A client-supplied idempotency key makes it safe to retry:

POST /v1/messages
Idempotency-Key: 8f2b1c0e-...       # chosen by the client, stable across its retries

The server stores the key with the request fingerprint and the eventual result:

  • Key unseen → run it, record (key → in-flight) before starting.
  • Key seen, in flight → return 409 Conflict, or block until the original finishes.
  • Key seen, completed → return the stored result without re-running.
  • Key seen, different request body → 422 — the client has a bug, and silently returning the old answer would hide it.

Records expire, typically after 24 hours, and the expiry must be documented because it is the window in which a retry is safe.

Errors the client can act on

DefinitionAn error taxonomy that tells the client what to do

Every error must answer two questions, because together they determine the client's next action:

Caller's faultService's fault
Terminal400 invalid request, 401, 403, 422501 not implemented
Retryable429 rate limited — with Retry-After503 overloaded, 504 deadline, 500 internal

A typed body carries the detail:

{ "type": "error",
  "error": { "type": "rate_limit_error",
             "message": "tokens-per-minute quota exhausted",
             "retry_after_ms": 1840 } }

Key idea

429 and 503 are the two that matter and they are routinely conflated. 429 means you asked for too much and the limit is yours; backing off fixes it. 503 means the service is out of capacity; backing off helps only if everyone does it, which is why it must be paired with jitter — see Reliability Patterns. Returning 500 for an overload turns a sheddable condition into one clients will hammer.

Versioning

HeuristicPin model versions, evolve schemas additively

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

Two independent things change and need separate versioning.

The wire schema evolves additively: new fields are optional, existing fields never change type or meaning, and nothing is removed within a major version. A date-based header (anthropic-version: 2023-06-01 is the shape) lets the server serve old semantics to old clients.

The model is named explicitly in the request, and the name must be pinnable:

"model": "claude-opus-4-5-20251101"     # pinned: byte-identical behaviour over time
"model": "claude-opus-4-5"              # alias: follows the latest point release

What this does not say

This page is about the synchronous request API. Long-running work — batch jobs, large document processing, asynchronous agent runs — needs a different shape entirely: submit returns a job id, the client polls or subscribes, and the result lives somewhere addressable. Trying to serve those through the streaming endpoint by raising timeouts is the usual mistake, and it fails at the first connection drop.

References

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page