THE FOLD / BOSS / THE GAUNTLET / THE CORRELATED FAILURE
THE CORRELATED FAILURE
an availability figure is a belief about shared fate
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
Three replicas at 99% give six nines — if they fail independently. They share a rack, a power feed, a kernel version and a deploy pipeline, so they do not.
LIT verified live. With a 1% single-replica failure rate and perfect independence, all three fail together with probability 0.000001: 6.00 nines. Fully correlated, they fail together with probability 0.01 — 2.00 nines, the same as one replica, and 10,000× worse. The collapse is not gradual: at a correlation of just 0.2 the figure is already 2.70 nines. Four of the six nines are gone by the time the replicas are one fifth alike.
LIT verified live. With a 1% single-replica failure rate and perfect independence, all three fail together with probability 0.000001: 6.00 nines. Fully correlated, they fail together with probability 0.01 — 2.00 nines, the same as one replica, and 10,000× worse. The collapse is not gradual: at a correlation of just 0.2 the figure is already 2.70 nines. Four of the six nines are gone by the time the replicas are one fifth alike.
2 HOW IT WAS WEAVED · AI + HUMAN
Correlated failure is why availability targets are written per failure domain, and why “multi-AZ” means something specific rather than “three copies”.
AVAN (AI) swept the correlation rather than contrasting the endpoints, because the endpoints suggest a trade-off and the sweep shows a cliff. Going from independent to 0.2 correlated costs more nines than going from 0.2 to fully correlated. The damage is done by the first small amount of shared fate.
AVAN (AI) swept the correlation rather than contrasting the endpoints, because the endpoints suggest a trade-off and the sweep shows a cliff. Going from independent to 0.2 correlated costs more nines than going from 0.2 to fully correlated. The damage is done by the first small amount of shared fate.
3 ONE DIMENSION
Correlation against nines. The cliff is at the left.
4 TWO DIMENSIONS · INTERACTIVE
Add replicas, or admit they share something.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: three copies of one fate.
AVAN’s addition (the inverse-companion): the forward reading is that correlation destroys the benefit of replication. The inverse is that the independence was never measured, only assumed. Nobody computes the correlation between their replicas; the six nines come from multiplying a number by itself three times, which is an arithmetic operation performed on an assumption. Read backwards, the availability figure in the design document is not a prediction about the system — it is a restatement of a belief about how much the replicas have in common, and that belief is usually never written down at all.
LIT a 1% single-replica failure rate with perfect independence gives all three failing together at probability 0.000001, which is 6.00 nines, while fully correlated they fail together at 0.01 - 2.00 nines, the same as one replica and 10,000 times worse - and the collapse is not gradual, since at a correlation of only 0.2 the figure is already 2.70 nines
FIG Correlated failure is why availability targets are written per failure domain and why multi-AZ means something specific rather than three copies. AVAN swept the correlation rather than contrasting the endpoints, because the endpoints suggest a trade-off and the sweep shows a cliff: going from independent to 0.2 correlated costs more nines than going from 0.2 to fully correlated. The damage is done by the first small amount of shared fate.
FIG Correlated failure is why availability targets are written per failure domain and why multi-AZ means something specific rather than three copies. AVAN swept the correlation rather than contrasting the endpoints, because the endpoints suggest a trade-off and the sweep shows a cliff: going from independent to 0.2 correlated costs more nines than going from 0.2 to fully correlated. The damage is done by the first small amount of shared fate.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE GAUNTLET · David Lee Wise (ROOT0), with AVAN