THE FOLD / BOSS / THE GAUNTLET / THE TAIL AT SCALE
THE TAIL AT SCALE
one in a hundred, a hundred times over
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
A service where only one request in a hundred is slow sounds healthy. Fan a single user request out to a hundred of those services and wait for all of them, and the rare event stops being rare — it becomes the common case.
LIT verified live. With 1% of leaf requests slow, a fan-out of 100 makes 63.40% of parent requests slow — exactly
LIT verified live. With 1% of leaf requests slow, a fan-out of 100 makes 63.40% of parent requests slow — exactly
1−0.99100. A 200,000-trial simulation gives 63.39%, off by 0.004 points. Half of all parents are slow at a fan-out of just 69. The leaf never got worse; only the arithmetic of waiting for all of them changed.2 HOW IT WAS WEAVED · AI + HUMAN
Jeff Dean and Luiz André Barroso set this out in The Tail at Scale (2013), and it is the reason large fan-out systems are engineered around tail latency rather than averages.
AVAN (AI) computed the exact figure and then simulated it as a separate check, because the closed form is easy to state and easy to mis-state. The number that reframes the problem is 69: you do not need a thousand-way fan-out for this to bite. A service with a 1-in-100 tail is already a coin flip at sixty-nine leaves, which is an ordinary page.
AVAN (AI) computed the exact figure and then simulated it as a separate check, because the closed form is easy to state and easy to mis-state. The number that reframes the problem is 69: you do not need a thousand-way fan-out for this to bite. A service with a 1-in-100 tail is already a coin flip at sixty-nine leaves, which is an ordinary page.
3 ONE DIMENSION
Fan-out against the chance the parent is slow.
4 TWO DIMENSIONS · INTERACTIVE
Widen the fan-out, or make the leaf better.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: one hundred leaves, one slow enough to matter.
AVAN’s addition (the inverse-companion): the forward reading is that fan-out amplifies the tail. The inverse is that fan-out did not amplify anything — it revealed what the average was hiding. The leaf service was always slow 1% of the time; that fact was simply never observable from a single call. Read backwards, scale is not a source of new failure modes so much as an instrument that finally has the resolution to see the old ones, and the alarming number is not 63.40% but the fact that 1% was ever considered a description of the service.
LIT with 1% of leaf requests slow, a fan-out of 100 makes 63.40% of parent requests slow - exactly 1 minus 0.99 to the hundredth - and a 200,000-trial simulation gives 63.39%, off by 0.004 points; half of all parents are slow at a fan-out of just 69, while the leaf never got worse
FIG Jeff Dean and Luiz Andre Barroso set this out in The Tail at Scale (2013), and it is why large fan-out systems are engineered around tail latency rather than averages. AVAN computed the exact figure and then simulated it as a separate check, because the closed form is easy to state and easy to mis-state. The number that reframes the problem is 69: a service with a 1-in-100 tail is already a coin flip at sixty-nine leaves, which is an ordinary page.
FIG Jeff Dean and Luiz Andre Barroso set this out in The Tail at Scale (2013), and it is why large fan-out systems are engineered around tail latency rather than averages. AVAN computed the exact figure and then simulated it as a separate check, because the closed form is easy to state and easy to mis-state. The number that reframes the problem is 69: a service with a 1-in-100 tail is already a coin flip at sixty-nine leaves, which is an ordinary page.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE GAUNTLET · David Lee Wise (ROOT0), with AVAN