Reliability Patterns
Timeouts, retries, jitter, circuit breakers, bulkheads and graceful degradation — and the arithmetic showing why an unbudgeted retry policy amplifies an outage.
Retries
Suppose each attempt fails independently with probability and is retried until it succeeds. The expected number of attempts per logical request is
so the load offered to the service is rather than . At that is ; at it is .
Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.
Cap retries globally rather than per request. Maintain a rolling ratio of retries to successful requests and refuse to retry beyond it:
class RetryBudget:
def __init__(self, ratio=0.1, window=60):
self.ratio, self.successes, self.retries = ratio, Rolling(window), Rolling(window)
def allow(self) -> bool:
if self.retries.sum() >= self.ratio * max(self.successes.sum(), 1):
metrics.incr("retry.budget_exhausted")
return False # the dependency is broken; retrying will not fix it
self.retries.add(1)
return True
With ratio , amplification is bounded by regardless of — a 10% budget means load never exceeds , where the unbounded policy would have reached .
Assumptions
A budget converts some retryable failures into user-visible errors, which is the point: when 90% of requests are failing, the marginal retry has a 10% chance of succeeding and a certainty of adding load. The budget encodes the judgement that at some failure rate the correct action is to stop and report, not to try harder.
Jitter
Suppose clients fail simultaneously — a dependency blip at time — and each retries after a deterministic backoff . All retries arrive in the instant , an offered rate of over any window of width . With full jitter, each client waits instead, and the expected number arriving in any window of width is — spread uniformly, with no spike.
# full jitter: pick uniformly below the exponential ceiling, not at it
def backoff(attempt, base=0.1, cap=20.0):
ceiling = min(cap, base * 2 ** attempt)
return random.uniform(0, ceiling)
Key idea
Exponential backoff without jitter solves the wrong half of the problem. It spreads one client's retries over time; it does nothing about many clients being synchronised with each other. Since what threatens the service is the aggregate, the randomisation is the part that matters — and "full jitter", sampling uniformly below the ceiling rather than jittering around it, is what measurement supports.
Circuit breakers
failures exceed threshold
┌────────┐ ──────────────────────► ┌────────┐
│ CLOSED │ │ OPEN │ fail fast, no calls
│ normal │ ◄────────────────────── └────────┘
└────────┘ probe succeeded │ after a cooldown
▲ ▼
│ ┌────────────┐
└──────────────────────────│ HALF-OPEN │ allow a few probes
probe succeeded └────────────┘
│ probe failed
└──► back to OPEN
Closed — calls pass; failures are counted over a rolling window. Open — calls fail immediately without touching the dependency, which both protects the caller's own resources and lets the dependency recover. Half-open — after a cooldown, a limited number of probes are allowed; success closes the circuit, failure reopens it.
Bulkheads
Partition every shared resource so that exhaustion in one place cannot starve another. The usual shared resources on an AI request path are connection pools, thread pools, GPU concurrency slots and — least obviously — the retry budget.
POOLS = {
"model": Semaphore(200), # GPU-bound work
"retrieval": Semaphore(50),
"tools": Semaphore(100), # third-party, least trustworthy
}
Without separation, a slow third-party tool holds every worker, and requests that never touch that tool fail too. With it, tool calls queue and time out while the rest of the service is unaffected.
Key idea
Bulkheads are the implementation of the pooling trade from Queueing and the Utilisation Wall, chosen deliberately in the other direction. Pooling minimises delay; partitioning minimises blast radius. You pay queueing delay to buy isolation, and the exchange rate is the pooling result — which tells you the partitioning is cheap when each partition still has several servers and expensive when partitions are small.
Degrading rather than failing
Under overload the goal is not to serve everything, it is to serve something to as many people as possible. The ladder, in the order it should be descended:
| Step | Action | Lost |
|---|---|---|
| 1 | serve from cache more aggressively (raise TTL, lower the semantic threshold) | freshness, some precision |
| 2 | reduce max_tokens, disable optional tool calls | answer completeness |
| 3 | route to a smaller, faster model | answer quality |
| 4 | drop retrieval, answer from the model alone | grounding |
| 5 | shed by priority — free tier first, paid last | availability for some |
| 6 | queue with a visible wait rather than failing | latency |
What this does not say
Every mechanism here is client-side or single-service. None of it addresses the correlated failure that Availability, Redundancy and Correlated Failure shows puts a floor under redundancy: a circuit breaker protects you from a dependency, not from a bad configuration pushed to every replica of your own service simultaneously. That is a deployment problem, and it is covered in Deployment and Rollout.
References
- BookB. Beyer, C. Jones, J. Petoff, N. R. Murphy (2016). Site Reliability Engineering: How Google Runs Production SystemsRetry budgets, load shedding and the cascading-failure analysis referenced here.
- Lecture notesM. Brooker (2015). Exponential Backoff and JitterThe measurement showing full jitter outperforms plain exponential backoff.
These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.
Results stated on this page
- PropositionRetries multiply load by 1/(1-f) at failure rate fproof includedWith unbounded retries and an independent failure probability f, each logical request generates 1/(1-f) attempts, which diverges as the service degrades.
- HeuristicBudget retries as a fraction of trafficCapping retries at a few percent of successful requests bounds amplification regardless of the failure rate.
- PropositionSynchronised backoff reproduces the spike it was avoidingproof includedWithout randomisation, clients that failed together retry together, so the retry traffic arrives as a spike rather than spread over the backoff interval.
- DefinitionThe circuit breakerA three-state machine that stops sending traffic to a failing dependency and probes it periodically rather than continuously.
- DefinitionBulkheadsPartitioning a shared resource per dependency or per tenant so that exhaustion by one cannot starve the others.