THE FOLD / BOSS / THE GATEKEEPER / THE GRAY FAILURE
THE GRAY FAILURE
the shallowness is restraint, not laziness
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
The worst kind of broken is the kind that answers the health check. A node that is slow, or failing only on the path your users take, stays in rotation because the thing watching it is not doing what they are doing.
LIT verified live. 200,000 samples of the same node. The health probe — small, cached, no dependencies — succeeds 99.91% of the time. Real requests, which touch the degraded dependency, succeed 61.93%. That is a gap of 37.99 points on one machine at one moment, and the node is not removed from rotation, because nothing that decides rotation ever saw the second number.
LIT verified live. 200,000 samples of the same node. The health probe — small, cached, no dependencies — succeeds 99.91% of the time. Real requests, which touch the degraded dependency, succeed 61.93%. That is a gap of 37.99 points on one machine at one moment, and the node is not removed from rotation, because nothing that decides rotation ever saw the second number.
2 HOW IT WAS WEAVED · AI + HUMAN
Gray failure was named by Huang et al. (HotOS 2017); the definition is precisely this differential observability between the system’s view and the user’s.
AVAN (AI) measured both populations rather than describing the idea, because the number that matters is the gap, not either rate. 99.91% is a true statement about the probe. 61.93% is a true statement about the users. Nothing is lying, and the node stays up.
AVAN (AI) measured both populations rather than describing the idea, because the number that matters is the gap, not either rate. 99.91% is a true statement about the probe. 61.93% is a true statement about the users. Nothing is lying, and the node stays up.
3 ONE DIMENSION
What the probe sees, and what the users see.
4 TWO DIMENSIONS · INTERACTIVE
Make the probe more like a real request.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: two observers, one machine.
AVAN’s addition (the inverse-companion): the forward reading is that shallow health checks miss real failures. The inverse is that a health check deep enough to catch this becomes the outage. A probe that exercises every dependency fails whenever any dependency is briefly slow, and a fleet that removes nodes on that signal removes all of them at once. Read backwards, the shallowness is not laziness — it is the thing stopping the health system from being the largest source of downtime, and gray failure is the price of that restraint.
LIT 200,000 samples of one node give a health probe - small, cached, no dependencies - succeeding 99.91% of the time while real requests touching the degraded dependency succeed 61.93%, a gap of 37.99 points on the same machine at the same moment, and the node is not removed from rotation because nothing that decides rotation saw the second number
FIG Gray failure was named by Huang et al. (HotOS 2017); the definition is precisely this differential observability between the system's view and the user's. AVAN measured both populations rather than describing the idea, because the number that matters is the gap, not either rate. 99.91% is a true statement about the probe and 61.93% is a true statement about the users - nothing is lying, and the node stays up.
FIG Gray failure was named by Huang et al. (HotOS 2017); the definition is precisely this differential observability between the system's view and the user's. AVAN measured both populations rather than describing the idea, because the number that matters is the gap, not either rate. 99.91% is a true statement about the probe and 61.93% is a true statement about the users - nothing is lying, and the node stays up.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE GATEKEEPER · David Lee Wise (ROOT0), with AVAN