AI Systems Engineering
Building the system around the model: gateways, rate limiting and quota, the Model Context Protocol, tool loops, semantic caching, GPU routing, cold starts, data pipelines, drift detection, evaluation harnesses and observability.
Level 1 — Core
- The Anatomy of an AI System6 min
Everything a request passes through besides the model: the components, what each one owns, and why the boundaries fall where they do.
The request pathControl plane and data planeOrder stages by cost of rejection - API Design for Model Endpoints6 min
Streaming, deadlines, cancellation, idempotency and versioning — the parts of a model API that are hard to change once anyone depends on them.
The streaming response contractPropagate deadlines, do not re-specify timeoutsIdempotency for non-idempotent generation
Level 2 — Advanced
- Gateways, Authentication and Quota6 min
What belongs in the gateway and what does not, how tenancy is resolved, and why quota and rate limiting are different mechanisms with different failure modes.
What a gateway ownsQuota and rate limit are different mechanismsThe identity chain - Rate Limiting Algorithms11 min
Token bucket, sliding window and GCRA, what each actually guarantees, and the two problems unique to limiting an LLM: metering tokens you have not generated yet, and doing it across many instances.
Token bucketWhat a token bucket guaranteesThe fixed window admits twice its limit - The Model Context Protocol8 min
What MCP standardises, the three server primitives and what controls each, the message shapes on the wire, and the security properties a host must enforce because the protocol cannot.
Hosts, clients and serversThe three server primitives and who controls eachCapability negotiation - Semantic Caching8 min
Answering a question you have answered before when it is not phrased the same way — the threshold that decides it, the cost of getting it wrong, and what must never be cached.
Three tiers of caching, doing three different jobsThe expected cost of a semantic cacheLowering the threshold trades precision for recall monotonically - Autoscaling and Cold Starts7 min
Why a GPU replica takes minutes to become useful, what that does to a scaling policy, and the four techniques that shorten it.
What the cold start is made ofA lower bound on weight loadingWhen scale-to-zero pays - Guardrails and Safety Layers3 min
Input and output checks around the model, what each can and cannot catch, and why the precision/recall trade is a product decision rather than a threshold.
Four places a check can sitRetrieved and tool content is untrustedA guardrail's cost is set by its false-positive rate at scale - Data Pipelines and Lineage6 min
The pipeline that feeds a model — ingestion, validation, chunking, embedding, indexing — and the lineage records that make "why did it answer that?" a question with an answer.
The stages, and what each may not doEvery stage must be idempotent and replayableThe lineage record - Reliability Patterns6 min
Timeouts, retries, jitter, circuit breakers, bulkheads and graceful degradation — and the arithmetic showing why an unbudgeted retry policy amplifies an outage.
Retries multiply load by 1/(1-f) at failure rate fBudget retries as a fraction of trafficSynchronised backoff reproduces the spike it was avoiding - Observability for AI Systems3 min
Tracing a request through prefill, tools and retrieval; accounting for tokens and cost per tenant; and keeping prompts out of your logs.
The span tree of a model requestWhat to record that a web service would notSample on the outcome, not on arrival - Vector Search Infrastructure4 min
Index types and what each trades, the recall/latency dial, filtering, freshness and sharding.
What each index type costsRetrieval recall caps end-to-end qualityThree ways to combine filters with vector search - Deployment and Rollout4 min
Shipping a model change: the registry, shadow traffic, canaries, and why a config change deserves more caution than a code change.
Six things whose change alters behaviourThe rollout ladderBuild the rollback before the rollout
Level 3 — Graduate
- Tool Loops in Production6 min
The agent loop as a piece of software: schema design, validation, parallelism, truncation, side effects and the termination conditions that stop it running forever.
The loopFour independent stopping conditionsTool results grow the context quadratically in cost - GPU Routing and Cache Affinity6 min
Choosing which replica serves a request when the replicas are not interchangeable — prefix affinity, queue-aware placement, LoRA multiplexing and draining.
The affinity–pooling tensionWhen to honour affinityRoute on outstanding work, not outstanding requests - Inference Server Architecture5 min
The software inside a replica: the scheduler loop, continuous batching, paged KV memory, preemption, and what disaggregating prefill from decode buys.
The iteration-level schedulerContiguous KV allocation wastes memory proportional to the length spreadTwo ways to reclaim a running sequence - Drift Detection9 min
What PSI actually measures, how to monitor an embedding distribution, and how to detect quality regressions when the labels arrive weeks late or never.
Four things called driftThe population stability indexPSI is the symmetrised KL divergence - Evaluation Harnesses7 min
Turning "is it good?" into a test suite that runs in CI — golden sets, graders, the failure modes of LLM-as-judge, and why a passing eval on a changed config proves nothing.
The four parts of an evalThe grader ladderModel judges have systematic biases