THE FOLD / RESPAWN / SECOND WIND / THE RETRY
THE RETRY
a measurement thrown away
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
Rerunning a failed test until it passes is a policy that cannot distinguish a flaky test from a broken one. Both of them go green, and only one of them should.
LIT verified live, closed form and simulation agreeing. A test that passes 40% of the time because the code is wrong is masked 40.00% of the time with no retry, 64.00% with one, 78.40% with two and 87.04% with three. Over 200,000 trials the simulation matches the closed form 1 − 0.6n+1 to within 0.06 points at every row. Three retries and the defect is invisible in seven runs out of eight.
LIT verified live, closed form and simulation agreeing. A test that passes 40% of the time because the code is wrong is masked 40.00% of the time with no retry, 64.00% with one, 78.40% with two and 87.04% with three. Over 200,000 trials the simulation matches the closed form 1 − 0.6n+1 to within 0.06 points at every row. Three retries and the defect is invisible in seven runs out of eight.
2 HOW IT WAS WEAVED · AI + HUMAN
Retry is genuinely the right answer for a test that depends on a network or a clock; the problem is that the policy is applied to the outcome and the outcome looks identical either way.
AVAN (AI) got the gate wrong here first — it demanded more than 90% masking at three retries when the arithmetic gives 87.04%. The measurement was right and the threshold was invented. Publishing the closed form beside the simulation is what makes that impossible to repeat.
AVAN (AI) got the gate wrong here first — it demanded more than 90% masking at three retries when the arithmetic gives 87.04%. The measurement was right and the threshold was invented. Publishing the closed form beside the simulation is what makes that impossible to repeat.
3 ONE DIMENSION
Retries against the chance a real defect goes green.
4 TWO DIMENSIONS · INTERACTIVE
Add retries. Watch the defect disappear.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: a failure given another go.
AVAN’s addition (the inverse-companion): the forward reading is that retries hide real defects. The inverse is that a retry is the only honest response to a test whose result is a random variable. If the outcome genuinely varies run to run, one observation is not evidence and re-running is basic method. Read backwards, the failure is not in the retry but in the reporting: the suite records “passed” where the truthful entry is “passed on attempt 3 of 4”, and every retry that is counted rather than logged is a measurement thrown away.
LIT a test that passes 40% of the time BECAUSE THE CODE IS WRONG is masked 40.00% of the time with no retry, 64.00% with one, 78.40% with two and 87.04% with three, and over 200,000 trials the simulation matches the closed form 1 - 0.6^(n+1) to within 0.06 points at every row - three retries and the defect is invisible in seven runs out of eight
FIG Retry is genuinely the right answer for a test that depends on a network or a clock; the problem is that the policy is applied to the outcome and the outcome looks identical either way. AVAN got the gate wrong here first - it demanded more than 90% masking at three retries when the arithmetic gives 87.04%. The measurement was right and the threshold was invented; publishing the closed form beside the simulation is what makes that impossible to repeat.
FIG Retry is genuinely the right answer for a test that depends on a network or a clock; the problem is that the policy is applied to the outcome and the outcome looks identical either way. AVAN got the gate wrong here first - it demanded more than 90% masking at three retries when the arithmetic gives 87.04%. The measurement was right and the threshold was invented; publishing the closed form beside the simulation is what makes that impossible to repeat.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of SECOND WIND · David Lee Wise (ROOT0), with AVAN