THE FOLD / RESPAWN / ROLLBACK / THE GOLDEN TEST
THE GOLDEN TEST
green means unchanged, never right
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
Record what the code does today, commit it, and fail the build when it changes. The suite now defends the current behaviour — including every bug in it.
LIT verified live. 8 golden files recorded from an implementation with a known defect. All 8 pass before the fix. Fix the defect and 2 of them fail — 25.0% of the suite goes red because the code became correct. The other 6 still pass. Green means unchanged; it has never meant right.
LIT verified live. 8 golden files recorded from an implementation with a known defect. All 8 pass before the fix. Fix the defect and 2 of them fail — 25.0% of the suite goes red because the code became correct. The other 6 still pass. Green means unchanged; it has never meant right.
2 HOW IT WAS WEAVED · AI + HUMAN
Golden or snapshot tests are enormously effective at catching unintended change, which is a different job from catching incorrectness, and the two are easy to confuse when the output is green.
AVAN (AI) reports the 6 that still pass alongside the 2 that break. A correct change breaking a quarter of the suite is the visible half; the invisible half is that three quarters of the golden files were never testing the defect at all and would have stayed green either way.
AVAN (AI) reports the 6 that still pass alongside the 2 that break. A correct change breaking a quarter of the suite is the visible half; the invisible half is that three quarters of the golden files were never testing the defect at all and would have stayed green either way.
3 ONE DIMENSION
Eight recordings, before and after the fix.
4 TWO DIMENSIONS · INTERACTIVE
Fix the bug. Re-record. Fix it again.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: yesterday, held as the answer.
AVAN’s addition (the inverse-companion): the forward reading is that golden tests freeze bugs into the expected output. The inverse is that they are the only tests that require no opinion about what is correct. Every other test needs somebody to know the right answer and write it down; a golden test needs only that the behaviour be observed, which is why they can be generated for systems nobody fully understands. Read backwards, the bug is frozen in because the method was designed for exactly the case where nobody could have said what should have been there instead.
LIT 8 golden files recorded from an implementation with a known defect all pass before the fix, and fixing the defect makes 2 of them fail - 25.0% of the suite goes red because the code became correct - while the other 6 still pass, so green means unchanged and has never meant right
FIG Golden or snapshot tests are enormously effective at catching unintended change, which is a different job from catching incorrectness, and the two are easy to confuse when the output is green. AVAN reports the 6 that still pass alongside the 2 that break: a correct change breaking a quarter of the suite is the visible half, and the invisible half is that three quarters were never testing the defect at all.
FIG Golden or snapshot tests are enormously effective at catching unintended change, which is a different job from catching incorrectness, and the two are easy to confuse when the output is green. AVAN reports the 6 that still pass alongside the 2 that break: a correct change breaking a quarter of the suite is the visible half, and the invisible half is that three quarters were never testing the defect at all.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of ROLLBACK · David Lee Wise (ROOT0), with AVAN