THE FOLD / RESPAWN / ROLLBACK / THE METASTABLE FAILURE
THE METASTABLE FAILURE
the only fix is refusing traffic you can serve
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
Some systems have two thresholds: the load at which they break, and the much lower load at which they recover. Between them the system stays broken with no cause present, sustained entirely by its own reaction to being broken.
LIT verified live. Capacity 1,000, retry amplification 3×. It breaks at an offered load of 1,010 — just past capacity, as expected. It recovers only when offered load falls to 330, because until then the retries alone exceed capacity. The gap is 680 requests, 68.0% of capacity: removing the trigger is not enough, and the load must drop to a third of what the system could originally serve.
LIT verified live. Capacity 1,000, retry amplification 3×. It breaks at an offered load of 1,010 — just past capacity, as expected. It recovers only when offered load falls to 330, because until then the retries alone exceed capacity. The gap is 680 requests, 68.0% of capacity: removing the trigger is not enough, and the load must drop to a third of what the system could originally serve.
2 HOW IT WAS WEAVED · AI + HUMAN
Metastable failure was characterised by Bronson et al. (HotOS 2021); the defining feature is exactly this sustaining effect that outlives its trigger.
AVAN (AI) computed both thresholds rather than describing hysteresis, because the gap is the whole phenomenon and it is a number. 1,010 to break, 330 to recover. An operator watching load return to normal sees a system that should be fine and is not, and the instinct — restart it, send the traffic back — puts it straight back over the line.
AVAN (AI) computed both thresholds rather than describing hysteresis, because the gap is the whole phenomenon and it is a number. 1,010 to break, 330 to recover. An operator watching load return to normal sees a system that should be fine and is not, and the instinct — restart it, send the traffic back — puts it straight back over the line.
3 ONE DIMENSION
Two thresholds, and the gap between them.
4 TWO DIMENSIONS · INTERACTIVE
Push it over, then try to bring it back.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: a state that holds itself down.
AVAN’s addition (the inverse-companion): the forward reading is that metastable failure is a system stuck below its capacity. The inverse is that it is a system doing its job perfectly at the wrong equilibrium. Nothing is degraded; every component is serving exactly as much as it can, and the work arriving is genuine work that genuine clients want done. Read backwards, there is no fault to find and nothing to repair — the only intervention that works is to refuse traffic you are able to serve, which is why the fix always feels wrong to the person who has to do it.
LIT with a capacity of 1,000 and a retry amplification of 3x the system breaks at an offered load of 1,010, just past capacity, but recovers only when offered load falls to 330, because until then the retries alone exceed capacity - a gap of 680 requests or 68.0% of capacity, so load must drop to a third of what the system could originally serve
FIG Metastable failure was characterised by Bronson et al. (HotOS 2021); the defining feature is exactly this sustaining effect that outlives its trigger. AVAN computed both thresholds rather than describing hysteresis, because the gap is the whole phenomenon and it is a number. An operator watching load return to normal sees a system that should be fine and is not, and the instinct - restart it, send the traffic back - puts it straight back over the line.
FIG Metastable failure was characterised by Bronson et al. (HotOS 2021); the defining feature is exactly this sustaining effect that outlives its trigger. AVAN computed both thresholds rather than describing hysteresis, because the gap is the whole phenomenon and it is a number. An operator watching load return to normal sees a system that should be fine and is not, and the instinct - restart it, send the traffic back - puts it straight back over the line.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of ROLLBACK · David Lee Wise (ROOT0), with AVAN