THE FOLD / LOOT / THE INVENTORY / THE DICTIONARY CODER
THE DICTIONARY CODER
incompressible is never a property of the data
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
There are two entirely different kinds of redundancy in a file: some symbols are more common than others, and some sequences repeat. An entropy coder sees only the first. A dictionary coder sees only the second.
LIT verified live. 64 uniformly random characters, repeated 200 times — 12,800 characters. Its order-0 entropy is 4.4086 bits per symbol, so an entropy coder reports 56,430 bits and declares the data nearly incompressible. A dictionary stores the unit once and a count: 520 bits. Same string, 109× apart, and both coders are working correctly.
LIT verified live. 64 uniformly random characters, repeated 200 times — 12,800 characters. Its order-0 entropy is 4.4086 bits per symbol, so an entropy coder reports 56,430 bits and declares the data nearly incompressible. A dictionary stores the unit once and a count: 520 bits. Same string, 109× apart, and both coders are working correctly.
2 HOW IT WAS WEAVED · AI + HUMAN
This is why DEFLATE is LZ77 followed by Huffman rather than either alone, and why compressing an already-compressed file achieves nothing.
AVAN (AI) built a string designed so the two measures disagree maximally, because the point is not that dictionaries are better. On a file with skewed symbol frequencies and no repeats the answer inverts exactly. 109× is not a ranking — it is the size of the blind spot each coder has, measured on a case chosen to expose one of them.
AVAN (AI) built a string designed so the two measures disagree maximally, because the point is not that dictionaries are better. On a file with skewed symbol frequencies and no repeats the answer inverts exactly. 109× is not a ranking — it is the size of the blind spot each coder has, measured on a case chosen to expose one of them.
3 ONE DIMENSION
One string, two verdicts.
4 TWO DIMENSIONS · INTERACTIVE
Trade repetition against skew and watch the winner swap.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: two blind spots, facing away.
AVAN’s addition (the inverse-companion): the forward reading is that you need both kinds of coder. The inverse is that “incompressible” is never a property of the data. It is a report from one particular model that found nothing it was built to look for, and the same bytes are 109× smaller to a model with a different appetite. Read backwards, every compression ratio is a statement about the compressor, and a file is only random with respect to whoever is looking at it.
LIT 64 uniformly random characters repeated 200 times - 12,800 characters - have an order-0 entropy of 4.4086 bits per symbol, so an entropy coder reports 56,430 bits and declares the data nearly incompressible, while a dictionary storing the unit once and a count needs 520 bits: the same string, 109 times apart, with both coders working correctly
FIG This is why DEFLATE is LZ77 followed by Huffman rather than either alone, and why compressing an already-compressed file achieves nothing. AVAN built a string designed so the two measures disagree maximally, because the point is not that dictionaries are better - on a file with skewed frequencies and no repeats the answer inverts exactly. 109x is not a ranking, it is the size of the blind spot each coder has.
FIG This is why DEFLATE is LZ77 followed by Huffman rather than either alone, and why compressing an already-compressed file achieves nothing. AVAN built a string designed so the two measures disagree maximally, because the point is not that dictionaries are better - on a file with skewed frequencies and no repeats the answer inverts exactly. 109x is not a ranking, it is the size of the blind spot each coder has.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE INVENTORY · David Lee Wise (ROOT0), with AVAN