THE FOLD / BOSS / THE WALL / THE CASCADING FAILURE
THE CASCADING FAILURE
no faulty component anywhere in it
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
A dependency loses capacity. Requests fail. Every failure is retried, so the offered load rises, so more fail, so more are retried. The retries are load, and the load is what caused the retries.
LIT verified live. Base load 1,000, healthy capacity 1,200, two retries per failure. Capacity drops to 600 for three rounds: offered load climbs to 3,400 with 2,800 failing. Capacity is then fully restored at round 5 — and by round 9 the offered load is 84,600, an amplification of 84.60×, with 83,400 still failing. The trigger is gone and the failure is not.
LIT verified live. Base load 1,000, healthy capacity 1,200, two retries per failure. Capacity drops to 600 for three rounds: offered load climbs to 3,400 with 2,800 failing. Capacity is then fully restored at round 5 — and by round 9 the offered load is 84,600, an amplification of 84.60×, with 83,400 still failing. The trigger is gone and the failure is not.
2 HOW IT WAS WEAVED · AI + HUMAN
Retry amplification is why every serious client library has a retry budget rather than a retry count, and why load shedding is applied at the caller.
AVAN (AI) set the healthy capacity above the base load at first, so nothing ever failed and there was no cascade to measure — a clean run proving nothing. The version that means something needs the dependency to actually lose capacity. Restoring it at round 5 is the important half: 84,600 against a base of 1,000, four rounds after the cause was removed.
AVAN (AI) set the healthy capacity above the base load at first, so nothing ever failed and there was no cascade to measure — a clean run proving nothing. The version that means something needs the dependency to actually lose capacity. Restoring it at round 5 is the important half: 84,600 against a base of 1,000, four rounds after the cause was removed.
3 ONE DIMENSION
Offered load per round. The capacity comes back at round 5.
4 TWO DIMENSIONS · INTERACTIVE
Change the retry policy and run it again.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: load feeding on its own failure.
AVAN’s addition (the inverse-companion): the forward reading is that retries cause cascading failure. The inverse is that every retry was individually correct. Each client saw a failed request and did the reasonable thing; not one of them misbehaved, and no single caller sent enough traffic to matter. Read backwards, this is a failure with no faulty component anywhere in it — the system is the defect, assembled entirely out of correct parts each doing what it was told.
LIT base load 1,000 against a healthy capacity of 1,200 with two retries per failure: when capacity drops to 600 for three rounds the offered load climbs to 3,400 with 2,800 failing, and after capacity is fully restored at round 5 the offered load reaches 84,600 by round 9 - an amplification of 84.60x with 83,400 still failing, four rounds after the cause was removed
FIG Retry amplification is why every serious client library has a retry budget rather than a retry count, and why load shedding is applied at the caller. AVAN set the healthy capacity above the base load at first, so nothing ever failed and there was no cascade to measure - a clean run proving nothing. Restoring capacity at round 5 is the important half: 84,600 against a base of 1,000, long after the trigger is gone.
FIG Retry amplification is why every serious client library has a retry budget rather than a retry count, and why load shedding is applied at the caller. AVAN set the healthy capacity above the base load at first, so nothing ever failed and there was no cascade to measure - a clean run proving nothing. Restoring capacity at round 5 is the important half: 84,600 against a base of 1,000, long after the trigger is gone.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE WALL · David Lee Wise (ROOT0), with AVAN