The Anatomy of an AI System

Everything a request passes through besides the model: the components, what each one owns, and why the boundaries fall where they do.

AI EngineeringCore⏱ 1 hour6 min readarchitecturerequest lifecyclecomponentsseparation of concerns

AssumesWhat a Serving System Must Guarantee

The path a request takes

DefinitionThe request path

The ordered stages between a client and a model, each owning one decision:

client
  │
  ▼
[1] edge / TLS termination        transport, compression, geography
  │
  ▼
[2] API gateway                   authenticate, resolve tenant, validate schema
  │
  ▼
[3] quota + rate limiter          admit, delay, or reject — before any cost is incurred
  │
  ▼
[4] semantic / exact cache        answer without the model, if we safely can
  │
  ▼
[5] router                        choose model, version, replica; consider KV affinity
  │
  ▼
[6] inference server              queue, batch, prefill, decode, stream
  │      ▲
  │      │ [7] tool / retrieval calls (MCP servers, vector search, functions)
  ▼      │
[8] post-processing               guardrails, redaction, schema validation
  │
  ▼
[9] telemetry + accounting        traces, token counts, cost attribution, evals
  │
  ▼
client

Stages 1–5 are cheap and fast; stage 6 is where essentially all the money is spent; stages 7–9 happen per request and must not become serial bottlenecks.

Key idea

The ordering is not arbitrary. Every stage before the inference server exists to avoid reaching the inference server, or to make reaching it cheaper. Authentication rejects strangers, the limiter rejects excess, the cache answers repeats, the router places the request where its KV cache already is. Each is orders of magnitude cheaper than a forward pass, so each one that fires is a large saving.

Which plane are you on

DefinitionControl plane and data plane

The data plane is everything on the per-request path: gateway, limiter, cache, router, inference servers. It must be fast, horizontally scalable, and survive the control plane being down.

The control plane is everything that configures the data plane: the model registry, the deployment controller, the quota configuration store, the feature-flag service, the autoscaler. It may be slower, may be a single region, and may be briefly unavailable.

Ordering the checks

HeuristicOrder stages by cost of rejection

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

Place the stages that can reject a request in increasing order of the work they consume, so a request that will be refused is refused as early as possible. Concretely, authentication before schema validation, schema validation before quota, quota before cache lookup, cache lookup before model dispatch.

Assumptions

This is a rule of thumb, not a theorem — it assumes the checks are independent and that any of them may reject. It fails when a later, more expensive check would reject far more often than an early cheap one; then the expected cost is minimised by running the high-rejection check first, which is the same reasoning as ordering predicates in a query plan by selectivity over cost.

Where state is allowed to live

HeuristicKeep the request path stateless except where state is the product

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

Only three kinds of per-request state genuinely need to persist across the call:

KV cache — lives on a specific GPU, is expensive to rebuild, and is the reason routing has affinity at all.
Conversation history — belongs in a store the client or the service owns explicitly, not implicitly in a server's memory.
Idempotency records — so a retried request returns the original result instead of running twice.

Everything else — limiter counters, cache entries, routing tables — should be reconstructible from the control plane or from scratch, so that any instance can be killed at any time.

Key idea

The KV cache is what makes an inference fleet different from a stateless web fleet. It gives requests affinity to a particular replica, and affinity is exactly what Queueing and the Utilisation Wall shows costs you queueing delay by re-partitioning a pooled queue. Every routing decision in this field is a negotiation between those two facts.

A minimal skeleton

The whole path, with each stage as a function that either returns a response or passes the request on. This is the shape the rest of the field fills in:

async def handle(req: Request) -> Response:
    # [2] who is this, and is the request well formed?
    tenant = await gateway.authenticate(req)          # raises 401
    body = gateway.validate(req, schema=ChatRequest)  # raises 400

    # [3] are they allowed to run this now?
    lease = await limiter.acquire(tenant, cost=estimate_tokens(body))
    if lease is None:
        raise TooManyRequests(retry_after=limiter.retry_after(tenant))

    # [4] can we answer without the model?
    if (hit := await cache.lookup(body)) is not None:
        telemetry.record(tenant, hit, cached=True)
        return hit.response

    # [5] where should this run?
    target = router.choose(body, affinity=body.conversation_id)

    # [6] the expensive part, streamed back as it is produced
    try:
        stream = await target.generate(body, deadline=req.deadline)
        response = await postprocess.guard(stream)     # [8]
    finally:
        await lease.release()                          # always, even on failure

    # [9] account for it whether or not it succeeded
    telemetry.record(tenant, response, cached=False)
    await cache.maybe_store(body, response)
    return response

What each later page covers

Stage	Page
2	API Design for Model Endpoints and Gateways, Authentication and Quota
3	Rate Limiting Algorithms
4	Semantic Caching
5	GPU Routing and Cache Affinity
6	Inference Server Architecture and Autoscaling and Cold Starts
7	The Model Context Protocol and Tool Loops in Production
8	Guardrails and Safety Layers
9	Observability for AI Systems

What this does not say

This page describes a single-model, single-region service. Multi-region adds a routing decision above stage 1 and a consistency question about where quota counters live; multi-model adds a capability-matching problem at stage 5. Both are compositions of what follows rather than new ideas, but neither is free, and a design that ignores them until it needs them usually discovers that the limiter and the router made region-local assumptions that are expensive to undo.

References

Lecture notesAnthropic (2024). Model Context Protocol — SpecificationThe protocol used for the tool and resource layer in later pages of this field.
BookB. Beyer, C. Jones, J. Petoff, N. R. Murphy (2016). Site Reliability Engineering: How Google Runs Production SystemsSource of the load-shedding, retry and error-budget practices referenced throughout this field.

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

DefinitionThe request pathThe ordered stages a request crosses between the client and the model, and the one job each stage owns.
DefinitionControl plane and data planeThe split between components on the per-request path and components that configure them.
HeuristicOrder stages by cost of rejectionPut the checks that can reject a request earliest, cheapest first, so that work is never spent on a request that will be refused.
HeuristicKeep the request path stateless except where state is the productOnly the KV cache and the conversation store hold per-request state worth keeping; everything else should be reconstructible.