THE FOLD / GLITCH / HEISENBUG / THE COORDINATED OMISSION
THE COORDINATED OMISSION
the meter went quiet exactly where the trouble was
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
A load generator that waits for each response cannot issue requests while the service is stalled. The requests that would have been sent during the stall are never sent, never timed, and never counted — so the worst moment in the run erases its own evidence.
LIT verified live. One service, one 100 ms stall, one intended rate of 1 request per ms for 10 seconds. The closed loop records 9,900 samples; the open loop records 10,000 — 100 requests simply vanished. The closed loop reports a p99.9 of 1 ms and exactly 1 sample over 10 ms. The open loop, same service, same stall, reports p99.9 of 92 ms and 91 samples over 10 ms.
LIT verified live. One service, one 100 ms stall, one intended rate of 1 request per ms for 10 seconds. The closed loop records 9,900 samples; the open loop records 10,000 — 100 requests simply vanished. The closed loop reports a p99.9 of 1 ms and exactly 1 sample over 10 ms. The open loop, same service, same stall, reports p99.9 of 92 ms and 91 samples over 10 ms.
2 HOW IT WAS WEAVED · AI + HUMAN
Gil Tene named coordinated omission and built HdrHistogram partly to make it visible; it is the reason a great many published latency numbers are wrong.
AVAN (AI) measured the shape of the lie rather than restating it. The p99 barely moves — 1 ms to 2 ms — because only 1% of the window is affected. It is the p99.9 that goes from 1 to 92. A benchmark that reports p99 and stops will show almost nothing wrong, which is precisely why the omission survives review: the number that would expose it is the one nobody printed.
AVAN (AI) measured the shape of the lie rather than restating it. The p99 barely moves — 1 ms to 2 ms — because only 1% of the window is affected. It is the p99.9 that goes from 1 to 92. A benchmark that reports p99 and stops will show almost nothing wrong, which is precisely why the omission survives review: the number that would expose it is the one nobody printed.
3 ONE DIMENSION
The same stall, seen by both meters.
4 TWO DIMENSIONS · INTERACTIVE
Change the stall and watch which percentile notices.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: a gap in the sampling, not in the service.
AVAN’s addition (the inverse-companion): the forward reading is that closed-loop benchmarks under-report latency. The inverse is that they report the truth about a system nobody has. A closed loop measures a world in which load politely stops arriving whenever you are struggling — and that world is real for exactly one user, the one holding the connection. Read backwards, coordinated omission is not a measurement bug; it is a faithful measurement of the wrong system, and the wrong system is the one the benchmark accidentally built.
LIT one service, one 100 ms stall, an intended rate of 1 request per ms for 10 seconds: the closed loop records 9,900 samples and the open loop 10,000, so 100 requests simply vanished; the closed loop reports a p99.9 of 1 ms and exactly 1 sample over 10 ms while the open loop, same service and same stall, reports p99.9 of 92 ms and 91 samples over 10 ms
FIG Gil Tene named coordinated omission and built HdrHistogram partly to make it visible; it is the reason a great many published latency numbers are wrong. AVAN measured the shape of the lie rather than restating it: the p99 barely moves, 1 ms to 2 ms, because only 1% of the window is affected, and it is the p99.9 that goes from 1 to 92. A benchmark that reports p99 and stops shows almost nothing wrong, which is precisely why the omission survives review.
FIG Gil Tene named coordinated omission and built HdrHistogram partly to make it visible; it is the reason a great many published latency numbers are wrong. AVAN measured the shape of the lie rather than restating it: the p99 barely moves, 1 ms to 2 ms, because only 1% of the window is affected, and it is the p99.9 that goes from 1 to 92. A benchmark that reports p99 and stops shows almost nothing wrong, which is precisely why the omission survives review.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of HEISENBUG · David Lee Wise (ROOT0), with AVAN