THE FOLD / GLITCH / RACE CONDITION / THE FLAKY TEST
THE FLAKY TEST
one is enough, and there are five thousand chances
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
A test that passes 99.9% of the time sounds excellent. Run five thousand of them and the suite is green less than one run in a hundred.
LIT verified live. At a per-test failure rate of 0.1%: 100 tests give a suite that passes 90.48% of the time. 1,000 tests: 36.77%. 5,000 tests: 0.67% green and 99.33% red, with 5.00 expected failures per run. Nothing about any individual test changed. Per-test reliability has to scale with the suite, and 99.9% does not survive contact with five thousand.
LIT verified live. At a per-test failure rate of 0.1%: 100 tests give a suite that passes 90.48% of the time. 1,000 tests: 36.77%. 5,000 tests: 0.67% green and 99.33% red, with 5.00 expected failures per run. Nothing about any individual test changed. Per-test reliability has to scale with the suite, and 99.9% does not survive contact with five thousand.
2 HOW IT WAS WEAVED · AI + HUMAN
This is why large suites quarantine flaky tests rather than tolerate them: at scale, tolerable individually is impossible collectively.
AVAN (AI) shows the 100-test row first because that is where the intuition is formed. 90.48% green feels like a slightly annoying suite. The same tests, fifty times as many, produce a suite that essentially never passes — and the only thing that changed is a count nobody thinks of as a reliability parameter.
AVAN (AI) shows the 100-test row first because that is where the intuition is formed. 90.48% green feels like a slightly annoying suite. The same tests, fifty times as many, produce a suite that essentially never passes — and the only thing that changed is a count nobody thinks of as a reliability parameter.
3 ONE DIMENSION
Suite size against the chance of a green run.
4 TWO DIMENSIONS · INTERACTIVE
Grow the suite. Fix the flake rate. Watch.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: five thousand near-certainties.
AVAN’s addition (the inverse-companion): the forward reading is that flaky tests make a suite unreliable. The inverse is that the suite is reporting correctly and the expectation is what is wrong. A red run at 5,000 tests and 0.1% flake is the accurate output of the system as built; demanding an all-green run is demanding 5,000 independent near-certainties to land at once. Read backwards, nobody chose that requirement — it arrived by adding tests, one at a time, each of which made the suite better and the green run less likely.
LIT at a per-test failure rate of 0.1%, 100 tests give a suite that passes 90.48% of the time, 1,000 tests give 36.77%, and 5,000 tests give 0.67% green against 99.33% red with 5.00 expected failures per run - nothing about any individual test changed, and 99.9% does not survive contact with five thousand
FIG This is why large suites quarantine flaky tests rather than tolerate them: at scale, tolerable individually is impossible collectively. AVAN shows the 100-test row first because that is where the intuition is formed. 90.48% green feels like a slightly annoying suite; the same tests, fifty times as many, produce a suite that essentially never passes, and the only thing that changed is a count nobody thinks of as a reliability parameter.
FIG This is why large suites quarantine flaky tests rather than tolerate them: at scale, tolerable individually is impossible collectively. AVAN shows the 100-test row first because that is where the intuition is formed. 90.48% green feels like a slightly annoying suite; the same tests, fifty times as many, produce a suite that essentially never passes, and the only thing that changed is a count nobody thinks of as a reliability parameter.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of RACE CONDITION · David Lee Wise (ROOT0), with AVAN