Availability, Redundancy and Correlated Failure

The arithmetic of nines — why chains multiply, why replicas help less than they look like they should, and the floor that correlated failure puts under any amount of redundancy.

AI SystemsAdvanced⏱ 1 hour7 min readavailabilityredundancyreliabilitycorrelated failurecommon mode

1 problem →

AssumesWhat a Serving System Must Guarantee · Probability Spaces and Random Variables

Series: nines erode

DefinitionAvailability

The availability $A$ of a component is the long-run fraction of requests (or of time) in which it is functioning. Its complement $U = 1-A$ is the unavailability. "Four nines" means $A = 0.9999$ , i.e. $U = 10^{-4}$ , i.e. about 53 minutes per year.

PropositionAvailability of a dependency chain

If a request requires $k$ components with independent availabilities $A_1,\dots,A_k$ , the end-to-end availability is

A_{\text{series}} \;=\; \prod_{i=1}^k A_i , \qquad\text{and}\qquad U_{\text{series}} \;=\; 1-\prod_i (1-U_i) \;\le\; \sum_{i=1}^k U_i ,

so unavailabilities add to first order.

Ten services each at three nines give $0.999^{10} = 0.99005$ — two nines, and about 87 hours of annual downtime. Depth in a call graph is expensive, and this is the quantitative case for collapsing dependencies on the critical path.

Parallel: nines accumulate

PropositionAvailability of independent replicas

If $n$ replicas each have availability $A$ and fail independently, and the system needs at least one,

A_{\text{parallel}} \;=\; 1-(1-A)^n \;=\; 1-U^n .

Two replicas at three nines give $1-10^{-6}$ — six nines. That is the calculation that makes redundancy look irresistible, and it is the one that is wrong in practice.

Quorums

PropositionQuorum availability

If the system requires at least $k$ of $n$ independent replicas,

A_{k/n} \;=\; \sum_{j=k}^{n}\binom nj A^j (1-A)^{n-j} .

This is decreasing in $k$ ; $k=1$ recovers the parallel formula and $k=n$ recovers the series formula $A^n$ .

The floor

TheoremCorrelated failure puts a floor under redundancy

Model failures with a common-cause fraction $\beta \in [0,1]$ : with probability $\beta$ a failure event takes down every replica simultaneously, and with probability $1-\beta$ replicas fail independently. Writing $U$ for a single replica's unavailability, the $n$ -replica system has

U_n \;=\; (1-\beta)\,U^{\,n} \;+\; \beta\,U ,

which is decreasing in $n$ with

\lim_{n\to\infty} U_n \;=\; \beta\,U \; > \;0 \quad\text{whenever } \beta > 0 .

No amount of redundancy reduces unavailability below $\beta U$ .

Assumptions

The beta-factor model is the simplest correlated-failure model: it splits failure causes into "independent" and "hits everything", with no intermediate structure. Real correlation is graded — a rack, a zone, a region — and is better modelled by a hierarchy. The theorem's conclusion survives any such refinement, because the coarsest shared level plays the role of $\beta$ .

Key idea

Take $A = 0.999$ , so $U = 10^{-3}$ , and suppose just 1% of failures are common-mode. Two replicas give

U_2 \;=\; 0.99\times10^{-6} + 0.01\times10^{-3} \;=\; 1.1\times10^{-5},

against $10^{-6}$ under the independence assumption — an order of magnitude worse — and the floor at $n=\infty$ is $10^{-5}$ . A 1% correlation costs more than the second replica buys. This is why availability work is dominated by decorrelation — staggered rollouts, separate control planes, different certificate expiries — rather than by adding replicas.

What this does not say

Availability is not latency. A replica that is up but slow counts as available in every formula here, while failing the SLO in What a Serving System Must Guarantee. Systems with generous timeouts convert latency failures into availability successes on the dashboard and into abandoned requests in reality. The honest measure is the fraction of requests served within the objective, which makes availability a special case of the latency SLO rather than a separate quantity.

References

PaperS. Gilbert, N. Lynch (2002). Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web ServicesThe formal statement and proof of the CAP theorem referenced at the end.
PaperK. N. Fleming (1975). A Reliability Model for Common Mode Failures in Redundant Safety SystemsOrigin of the beta-factor model for common-cause failure used in the final theorem.

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

PropositionAvailability of a dependency chainproof includedA request needing k independent services succeeds with probability equal to the product of their availabilities, so unavailabilities approximately add.
PropositionAvailability of independent replicasproof includedWith n independent replicas and any one sufficing, unavailability is the n-th power of a single replica's, so each replica multiplies the nines.
PropositionQuorum availabilityproof includedA system needing k of n replicas is available with probability given by a binomial tail, which is maximised at k = 1 and degrades sharply once k exceeds n/2.
TheoremCorrelated failure puts a floor under redundancyproof includedIf a fraction beta of failures are common-mode, unavailability cannot be driven below beta times a single component's unavailability, however many replicas are added.
DefinitionAvailabilityThe long-run fraction of requests a component serves successfully, and its complement, unavailability, measured in nines.

Problems (1)

All problems

The floor that correlated failure puts under redundancyProofIntermediate⏱ 30 min