THE FOLD / BOSS / THE GATEKEEPER / THE ORACLE PROBLEM
THE ORACLE PROBLEM
it compares two attempts at the same belief
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
To test something you need to know what the answer should be. For most of what a program computes, nobody does — and every technique for testing at scale is a way of avoiding having to.
LIT verified live. 5 strategies exercising 1,106,012 inputs between them. Only 6,012 come with a stated expected value: 0.544%. 3 strategies have an oracle — hand-computed cases, round-tripping through an inverse, and comparison against a slow reference. 2 have none: invariants check that something holds without knowing the value, and production traffic is never scored at all.
LIT verified live. 5 strategies exercising 1,106,012 inputs between them. Only 6,012 come with a stated expected value: 0.544%. 3 strategies have an oracle — hand-computed cases, round-tripping through an inverse, and comparison against a slow reference. 2 have none: invariants check that something holds without knowing the value, and production traffic is never scored at all.
2 HOW IT WAS WEAVED · AI + HUMAN
This is the test oracle problem, and it is why metamorphic testing exists: if you cannot say what f(x) is, you may still be able to say that f(2x) should be twice it.
AVAN (AI) counts inputs rather than tests, because the inputs are where the asymmetry is. Adding a million more inputs to the invariant strategy moves the exercised count enormously and the checked count not at all — more testing, in a sense that does not include more knowing.
AVAN (AI) counts inputs rather than tests, because the inputs are where the asymmetry is. Adding a million more inputs to the invariant strategy moves the exercised count enormously and the checked count not at all — more testing, in a sense that does not include more knowing.
3 ONE DIMENSION
Five strategies. Two of them never score anything.
4 TWO DIMENSIONS · INTERACTIVE
Add inputs and watch the checked fraction fall.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: an answer nobody can state.
AVAN’s addition (the inverse-companion): the forward reading is that most inputs go unchecked because no oracle exists. The inverse is that an oracle you do have is a second implementation, and it can be the wrong one. A slow reference, an inverse function, a hand-computed table — each is code or judgement that can itself be mistaken, and when they disagree with the system under test nothing says which one is right. Read backwards, testing does not compare a program to the truth; it compares two attempts at the same belief, and calls the older one correct.
LIT 5 strategies exercising 1,106,012 inputs between them leave only 6,012 with a stated expected value - 0.544% - because 3 strategies have an oracle in hand-computed cases, round-tripping through an inverse and comparison against a slow reference, while 2 have none: invariants check that something holds without knowing the value, and production traffic is never scored at all
FIG This is the test oracle problem, and it is why metamorphic testing exists: if you cannot say what f(x) is, you may still be able to say that f(2x) should be twice it. AVAN counts inputs rather than tests, because the inputs are where the asymmetry is - adding a million more to the invariant strategy moves the exercised count enormously and the checked count not at all.
FIG This is the test oracle problem, and it is why metamorphic testing exists: if you cannot say what f(x) is, you may still be able to say that f(2x) should be twice it. AVAN counts inputs rather than tests, because the inputs are where the asymmetry is - adding a million more to the invariant strategy moves the exercised count enormously and the checked count not at all.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE GATEKEEPER · David Lee Wise (ROOT0), with AVAN