THE FOLD / CO-OP / THE SYNC / THE TEST ORDERING
THE TEST ORDERING
the bill for a speed-up nobody wrote down
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
Tests that share state are not tests of the code — they are tests of the code plus whatever ran before them. A green suite may be one arrangement out of many, and the runner picked it.
LIT verified live, exhaustively. 3 tests over one piece of shared state. Of the 6 possible orderings, 2 pass and 4 fail — a pass rate of 33.3%. The two that work are ABC and CAB. Nothing is wrong with any individual test, nothing is wrong with the code, and two thirds of the ways of running them report a failure.
LIT verified live, exhaustively. 3 tests over one piece of shared state. Of the 6 possible orderings, 2 pass and 4 fail — a pass rate of 33.3%. The two that work are ABC and CAB. Nothing is wrong with any individual test, nothing is wrong with the code, and two thirds of the ways of running them report a failure.
2 HOW IT WAS WEAVED · AI + HUMAN
This is why serious runners randomise order and why a seed appears in the output — a suite that only passes in declaration order has a defect that declaration order is hiding.
AVAN (AI) enumerated all 6 rather than sampling, because the interesting quantity is which orderings work and there are few enough to say. ABC and CAB is not a pattern anybody would predict from reading the tests.
AVAN (AI) enumerated all 6 rather than sampling, because the interesting quantity is which orderings work and there are few enough to say. ABC and CAB is not a pattern anybody would predict from reading the tests.
3 ONE DIMENSION
Six orderings, two of them green.
4 TWO DIMENSIONS · INTERACTIVE
Run an ordering and watch the shared state.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: an arrangement that happened to work.
AVAN’s addition (the inverse-companion): the forward reading is that order-dependent tests are broken tests. The inverse is that the dependency is usually the only thing making the suite affordable. Building the fixture once and letting twenty tests share it is why the suite finishes in seconds; isolating each one means paying full setup twenty times. Read backwards, order dependence is not carelessness — it is the residue of a performance decision, and the 4 failing orderings are the bill for a speed-up nobody wrote down as a trade.
LIT 3 tests over one piece of shared state give 6 possible orderings of which 2 pass and 4 fail, a pass rate of 33.3%, and the two that work are ABC and CAB - nothing is wrong with any individual test, nothing is wrong with the code, and two thirds of the ways of running them report a failure
FIG This is why serious runners randomise order and why a seed appears in the output - a suite that only passes in declaration order has a defect that declaration order is hiding. AVAN enumerated all 6 rather than sampling, because the interesting quantity is which orderings work and there are few enough to say. ABC and CAB is not a pattern anybody would predict from reading the tests.
FIG This is why serious runners randomise order and why a seed appears in the output - a suite that only passes in declaration order has a defect that declaration order is hiding. AVAN enumerated all 6 rather than sampling, because the interesting quantity is which orderings work and there are few enough to say. ABC and CAB is not a pattern anybody would predict from reading the tests.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE SYNC · David Lee Wise (ROOT0), with AVAN