THE FOLD / LOOT / THE VAULT / THE MUTATION SCORE
THE MUTATION SCORE
the fraction of a particular list
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
Change the code on purpose, in small wrong ways, and see whether the suite notices. What it fails to notice is the honest measure of what it was checking.
LIT verified live. 8 mutants against a suite with 100% line coverage. 6 are killed and 2 survive: a mutation that deletes the entire bonus, and one that changes nothing at all. Mutation score 75.0% where coverage says 100. 1 of the survivors is equivalent — it produces identical output on all 8 inputs, so no test can ever kill it and the score can never reach 100.
LIT verified live. 8 mutants against a suite with 100% line coverage. 6 are killed and 2 survive: a mutation that deletes the entire bonus, and one that changes nothing at all. Mutation score 75.0% where coverage says 100. 1 of the survivors is equivalent — it produces identical output on all 8 inputs, so no test can ever kill it and the score can never reach 100.
2 HOW IT WAS WEAVED · AI + HUMAN
Mutation testing is Lipton, DeMillo and Sayward’s (1978); the equivalent-mutant problem is undecidable in general, which is why the score is always a lower bound.
AVAN (AI) planted an equivalent mutant deliberately. Without it the number reads like a target to push to 100. With it, the ceiling is unreachable and unknown, and the score becomes what it actually is — a comparison against yesterday rather than a percentage of anything.
AVAN (AI) planted an equivalent mutant deliberately. Without it the number reads like a target to push to 100. With it, the ceiling is unreachable and unknown, and the score becomes what it actually is — a comparison against yesterday rather than a percentage of anything.
3 ONE DIMENSION
Eight mutants, six killed.
4 TWO DIMENSIONS · INTERACTIVE
Add the test that kills a survivor.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: a fault nobody would notice.
AVAN’s addition (the inverse-companion): the forward reading is that mutation testing measures suite quality honestly. The inverse is that it measures the suite against faults somebody thought to inject. The mutation operators are a catalogue — swap an operator, delete a statement, flip a boundary — and a real defect that is not in the catalogue leaves the score untouched. Read backwards, the score is not the fraction of bugs the suite would catch; it is the fraction of a particular list, and its value comes from the list being longer than the one in your head.
LIT 8 mutants against a suite with 100% line coverage leave 6 killed and 2 surviving - a mutation that deletes the entire bonus, and one that changes nothing at all - for a mutation score of 75.0% where coverage says 100, and 1 of the survivors is EQUIVALENT, producing identical output on all 8 inputs so no test can ever kill it and the score can never reach 100
FIG Mutation testing is Lipton, DeMillo and Sayward's (1978); the equivalent-mutant problem is undecidable in general, which is why the score is always a lower bound. AVAN planted an equivalent mutant deliberately. Without it the number reads like a target to push to 100. With it the ceiling is unreachable and unknown, and the score becomes what it actually is - a comparison against yesterday rather than a percentage of anything.
FIG Mutation testing is Lipton, DeMillo and Sayward's (1978); the equivalent-mutant problem is undecidable in general, which is why the score is always a lower bound. AVAN planted an equivalent mutant deliberately. Without it the number reads like a target to push to 100. With it the ceiling is unreachable and unknown, and the score becomes what it actually is - a comparison against yesterday rather than a percentage of anything.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE VAULT · David Lee Wise (ROOT0), with AVAN