THE INTEGER PROMOTION

JavaScript numbers are 64-bit floats until you use a bitwise operator, at which point they are silently converted to signed 32-bit integers and back. No error, no warning.

THE INTEGER PROMOTION · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · GLITCH · OFF BY ONE ◆ .dlw.fold THE FOLD / GLITCH / OFF BY ONE / THE INTEGER PROMOTION THE INTEGER PROMOTION the truncation is the only honest part 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION JavaScript numbers are 64-bit floats until you use a bitwise operator, at which point they are silently converted to signed 32-bit integers, operated on, and converted back. The conversion is not an error and produces no warning. LIT verified live. 1 << 31 is −2147483648 : shifting a positive number left makes it negative. 1 << 32 is 1 , because the shift count itself wraps at 32. And 2 32 | 0 is 0 — four billion becomes nothing. Of the 64 powers of two tested, 33 change value when passed through a single | 0 . 2 HOW IT WAS WEAVED · AI + HUMAN The ToInt32 conversion is specified in ECMA-262 and is the reason |0 was used as an optimisation hint for years by people who understood exactly what it truncates. AVAN (AI) swept every power of two rather than showing the famous shift, because the interesting boundary is 2 30 : everything at or below survives, everything above does not, and there is no diagnostic anywhere. 33 of 64 is the shape of a cliff sitting in the middle of the number line with no fence around it. 3 ONE DIMENSION Every power of two, before and after a bitwise operator. 4 TWO DIMENSIONS · INTERACTIVE Walk across the boundary. next power ▶ jump to the cliff reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a 64-bit value passing through a 32-bit door. AVAN’s addition (the inverse-companion): the forward reading is that bitwise operators truncate silently and that is a hazard. The inverse is that the truncation is the only honest part . A double can hold integers up to 2 53 and pretends to be an integer type the whole way; the bitwise operator is the single place the language admits there is a width at all. Read backwards, the surprise is not that |0 narrows — it is that everything else let you believe the number had no size. pause spin LIT 1 FIG The ToInt32 conversion is specified in ECMA-262 and is why |0 was used as an optimisation hint for years by people who understood exactly what it truncates. AVAN swept every power of two rather than showing the famous shift, because the interesting boundary is 2^30: everything at or below survives, everything above does not, and there is no diagnostic anywhere. 33 of 64 is the shape of a cliff in the middle of the number line with no fence around it. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of OFF BY ONE · David Lee Wise (ROOT0), with AVAN


THE MODULAR BIAS

Take a uniform random number in [0, R) and reduce it modulo k. Unless k divides R exactly, some residues get one extra chance and the result is not uniform.

I-13 v2.0 | net = binds - k | 4 planes, 13 symbols, 12 operants, 5 cortex rules | sha 64881ebf

THE MODULAR BIAS · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · LOOT · THE MINT ◆ .dlw.fold THE FOLD / LOOT / THE MINT / THE MODULAR BIAS THE MODULAR BIAS you cannot partition a set into equal parts that do not exist 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Take a uniform random number in [0, R) and reduce it modulo k . Unless k divides R exactly, some residues get one extra chance and the result is not uniform. LIT verified live and computed exactly, not sampled. Over R = 2 32 : with k = 6 there are 715,827,882 complete cycles and 4 favoured residues, an excess of 0.00000014% — harmless. With k = 1,000,000,007 there are only 4 complete cycles and the excess is 25% . With k = 3,000,000,000 there is 1 cycle and the excess is 100% : some outcomes are twice as likely as others. 2 HOW IT WAS WEAVED · AI + HUMAN Modulo bias is why every good library uses rejection sampling rather than rand() % n . AVAN (AI) had the relationship backwards at first. I gated on a large divisor giving negligible bias, which is the intuition and is wrong: a large k leaves fewer complete cycles in the range, so the bias grows with k/R . Correcting it produced the better instrument — the sweep from 715,827,882 cycles down to 1 is the whole mechanism, visible in a single column. 3 ONE DIMENSION Divisor against excess probability. Exact, not sampled. 4 TWO DIMENSIONS · INTERACTIVE Grow the divisor until the bias is unmissable. bigger divisor ▶ smaller reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a range that does not divide evenly. AVAN’s addition (the inverse-companion): the forward reading is that modulo introduces bias into a uniform source. The inverse is that the source was never uniform over the thing you wanted . It was uniform over 2 32 outcomes, and you asked for k ; unless k divides that, no function of one draw can be uniform, because you cannot partition a set into equal parts that do not exist. Read backwards, rejection sampling is not a correction — it is the admission that sometimes you must throw the draw away and ask again, because there is no arithmetic that makes the counts come out even. pause spin LIT computed exactly rather than sampled over R = 2^32: k = 6 leaves 715,827,882 complete cycles and 4 favoured residues for an excess of 0.00000014%, k = 1,000,000,007 leaves only 4 cycles and an excess of 25%, and k = 3,000,000,000 leaves 1 cycle and an excess of 100% - some outcomes twice as likely as others FIG Modulo bias is why every good library uses rejection sampling rather than rand() % n. AVAN had the relationship backwards at first: I gated on a large divisor giving negligible bias, which is the intuition and is wrong, because a large k leaves fewer complete cycles in the range so the bias grows with k/R. Correcting it produced the better instrument - the sweep from 715,827,882 cycles down to 1 is the whole mechanism in a single column. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE MINT · David Lee Wise (ROOT0), with AVAN


THE FLOAT EQUALITY

There are at least three notions of the same number in floating point, and they disagree with each other by design rather than by accident.

THE FLOAT EQUALITY · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · BOSS · THE WALL ◆ .dlw.fold THE FOLD / BOSS / THE WALL / THE FLOAT EQUALITY THE FLOAT EQUALITY a tolerance moves the uncertainty into a constant nobody revisits 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION There are at least three notions of “the same number” in floating point, and they disagree with each other by design rather than by accident. LIT verified live. NaN === NaN is false . 0 === -0 is true , but 1/0 === 1/-0 is false and Object.is(0, -0) is false — the same two values, three verdicts. Across 200,000 near-pairs the implication runs one way only: equality forces epsilon-closeness, with 0 counterexamples, while 106,390 pairs are within 1e-9 and are not equal. 2 HOW IT WAS WEAVED · AI + HUMAN The three notions are IEEE 754 equality, bitwise identity, and application tolerance; only the first two are specified anywhere. AVAN (AI) corrected how the zero was reported. My harness printed “equal but not within epsilon: 0 ” as though it were a measurement that came out empty. It is not — a === b forces |a-b| to be exactly zero, so that count can never be anything else. Reported as a count it looks like a near miss; reported as a one-way implication it is the actual structure. 3 ONE DIMENSION Five comparisons, and what each operator says. 4 TWO DIMENSIONS · INTERACTIVE Move the tolerance and watch the two tests separate. looser epsilon ▶ tighter reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: one question, three answers. AVAN’s addition (the inverse-companion): the forward reading is that float equality is unreliable and you should compare with a tolerance. The inverse is that the tolerance is a claim about your problem that the numbers cannot check . === is fully specified and always right about what it was asked; 1e-9 is a guess about how much difference matters here, and it will be wrong at some scale in the same program. Read backwards, replacing equality with a tolerance does not remove the uncertainty — it moves it out of the standard and into a constant nobody will revisit. pause spin LIT NaN === NaN is false, 0 === -0 is true but 1/0 === 1/-0 is false and Object.is(0,-0) is false - the same two values, three verdicts - and across 200,000 near-pairs the implication runs one way only: equality forces epsilon-closeness with 0 counterexamples, while 106,390 pairs are within 1e-9 and are not equal FIG The three notions are IEEE 754 equality, bitwise identity, and application tolerance; only the first two are specified anywhere. AVAN corrected how the zero was reported: my harness printed equal-but-not-within-epsilon as 0 as though it were a measurement that came out empty, when a === b forces the difference to be exactly zero, so that count can never be anything else. Reported as a count it looks like a near miss; as a one-way implication it is the actual structure. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE WALL · David Lee Wise (ROOT0), with AVAN


THE DOUBLE ROUNDING

Round to three decimals, then to one, and you sometimes get a different answer than rounding straight to one - because the first rounding can push a value across the boundary the second is looking at.

THE DOUBLE ROUNDING · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · GRIND · THE GRINDSTONE ◆ .dlw.fold THE FOLD / GRIND / THE GRINDSTONE / THE DOUBLE ROUNDING THE DOUBLE ROUNDING correctness does not compose 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Round a value to three decimals, then round that to one. You will sometimes get a different answer than rounding the original straight to one — because the first rounding can push a value across the boundary the second one is looking at. LIT verified live. 1,000,000 values swept from 0 to 10. Rounding once to one decimal and rounding twice via three decimals disagree 4,996 times — 0.4996% . At 0.0495 , rounding once gives 0 and rounding twice gives 0.1 : the intermediate step turned it into 0.05, which then rounded up. 2 HOW IT WAS WEAVED · AI + HUMAN Double rounding is why x87’s 80-bit intermediates were a correctness problem rather than a bonus, and why IEEE 754 specifies a single correctly-rounded result for each operation. AVAN (AI) swept a million values rather than presenting the boundary case, because 0.4996% is the useful number: roughly one value in two hundred. It is rare enough to survive testing and common enough to appear in production, which is the worst possible frequency for a defect. 3 ONE DIMENSION Where the two paths disagree, across the sweep. 4 TWO DIMENSIONS · INTERACTIVE Walk to a disagreement and watch the intermediate move it. next disagreement ▶ wider intermediate reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a value crossing a line it had not reached. AVAN’s addition (the inverse-companion): the forward reading is that intermediate precision causes double rounding errors. The inverse is that the extra precision was correct at every step . Rounding to three decimals is right; rounding that to one is right; the composition is wrong, and no individual operation misbehaved. Read backwards, correctness does not compose — a chain of individually correct roundings is not a correct rounding, and that is why the standard specifies results rather than steps. pause spin LIT 1,000,000 values swept from 0 to 10 give 4,996 disagreements between rounding once to one decimal and rounding twice via three - 0.4996%, roughly one value in two hundred - and at 0.0495 rounding once gives 0 while rounding twice gives 0.1, because the intermediate step turned it into 0.05 which then rounded up FIG Double rounding is why x87's 80-bit intermediates were a correctness problem rather than a bonus, and why IEEE 754 specifies a single correctly-rounded result for each operation. AVAN swept a million values rather than presenting the boundary case, because 0.4996% is the useful number: rare enough to survive testing and common enough to appear in production, which is the worst possible frequency for a defect. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE GRINDSTONE · David Lee Wise (ROOT0), with AVAN


THE VARINT

Most integers are small. A fixed 32-bit field spends four bytes on the number seven. A varint spends seven bits per byte on the value and one on the question is there more.

THE VARINT · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · LOOT · THE STASH ◆ .dlw.fold THE FOLD / LOOT / THE STASH / THE VARINT THE VARINT the wasted bytes were buying the ability to not look 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Most integers are small. A fixed 32-bit field spends four bytes on the number seven. A varint spends seven bits per byte on the value and one on the question “is there more?” LIT verified live. 200,000 values encoded and decoded with 0 round-trip failures. The boundaries are exact: 1 byte up to 127 , 2 up to 16,383 , 3 up to 2,097,151 . Across the range the mean is 2.917 bytes against a fixed 4 — 27.06% smaller, and that is the worst case for varints because the values are uniformly spread. 2 HOW IT WAS WEAVED · AI + HUMAN Varints are the backbone of Protocol Buffers and appear in nearly every binary format that expects small numbers. AVAN (AI) chose a uniform sweep deliberately, which is the least flattering input possible. Real data is skewed toward small values and does far better; quoting that figure would be quoting the marketing. 27.06% on uniform values is the floor, and a floor is worth more than a best case. 3 ONE DIMENSION Bytes per value, and where each boundary sits. 4 TWO DIMENSIONS · INTERACTIVE Encode a value and watch the continuation bits. bigger value ▶ smaller next boundary reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a length that is part of the value. AVAN’s addition (the inverse-companion): the forward reading is that varints make small numbers cheap. The inverse is that they make every number unskippable . A fixed field can be jumped over without being read; a varint must be decoded byte by byte to find out where it ends, so random access into an array of them is gone. Read backwards, the four wasted bytes were buying you the ability to not look, and a format that always knows where things are is paying for that with space. pause spin LIT 200,000 values encode and decode with 0 round-trip failures at exact boundaries - 1 byte up to 127, 2 up to 16,383, 3 up to 2,097,151 - for a mean of 2.917 bytes against a fixed 4, which is 27.06% smaller and is the worst case for varints because the values are uniformly spread FIG Varints are the backbone of Protocol Buffers and appear in nearly every binary format that expects small numbers. AVAN chose a uniform sweep deliberately, which is the least flattering input possible: real data is skewed toward small values and does far better, and quoting that figure would be quoting the marketing. 27.06% on uniform values is the floor, and a floor is worth more than a best case. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE STASH · David Lee Wise (ROOT0), with AVAN


THE ZIGZAG

A varint assumes small numbers are cheap, and two's complement makes -1 the largest number there is. Zigzag interleaves positive and negative so small magnitudes stay small on either side of zero.

THE ZIGZAG · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · LOOT · THE VAULT ◆ .dlw.fold THE FOLD / LOOT / THE VAULT / THE ZIGZAG THE ZIGZAG a subsidy paid by positives to rescue negatives 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION A varint assumes small numbers are cheap, and two’s complement makes −1 the largest number there is. Zigzag interleaves positive and negative so that small magnitudes stay small whichever side of zero they are on. LIT verified live. Every value from −100,000 to 100,000 — 200,001 of them — maps and maps back with 0 failures: the encoding is a bijection, not an approximation. −1 becomes 1 and costs 1 byte, where the two’s-complement bit pattern would cost 10 . That is 9 bytes saved on the most common small negative there is. 2 HOW IT WAS WEAVED · AI + HUMAN Zigzag is the sint32 and sint64 types in Protocol Buffers, and it exists solely because varints and two’s complement disagree about what “small” means. AVAN (AI) checked bijectivity across the whole range rather than spot-checking the mapping, because an encoding that is almost reversible is worthless. 200,001 of 200,001 is the claim; the byte counts are the payoff, and the interesting one is that zero costs the same either way — the saving is entirely on the negative side. 3 ONE DIMENSION The interleave, and what each value costs. 4 TWO DIMENSIONS · INTERACTIVE Cross zero and watch the cost stay flat. step up ▶ step down jump negative reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a number line folded at zero. AVAN’s addition (the inverse-companion): the forward reading is that zigzag makes negative numbers cheap. The inverse is that it makes the sign bit expensive for everyone . Every positive value now costs one extra bit of magnitude, because the low bit has been given to the sign — so the encoding is a subsidy paid by common positives to rescue common negatives. Read backwards, this is a bet about your data, and if your values are never negative it is a bet you lose on every single one. pause spin LIT every value from -100,000 to 100,000 - 200,001 of them - maps and maps back with 0 failures, so the encoding is a bijection rather than an approximation, and -1 becomes 1 costing 1 byte where the two's-complement bit pattern would cost 10, saving 9 bytes on the most common small negative there is FIG Zigzag is the sint32 and sint64 types in Protocol Buffers, and it exists solely because varints and two's complement disagree about what small means. AVAN checked bijectivity across the whole range rather than spot-checking the mapping, because an encoding that is almost reversible is worthless. 200,001 of 200,001 is the claim; the interesting payoff is that zero costs the same either way, so the saving is entirely on the negative side. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE VAULT · David Lee Wise (ROOT0), with AVAN


THE FRAME OF REFERENCE

A column of timestamps looks like large numbers and is really a small range with a big offset. Store the minimum once and the distances from it.

THE FRAME OF REFERENCE · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · LOOT · THE INVENTORY ◆ .dlw.fold THE FOLD / LOOT / THE INVENTORY / THE FRAME OF REFERENCE THE FRAME OF REFERENCE it compresses nothing and merely stops repeating yourself 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION A column of timestamps looks like large numbers and is really a small range with a big offset. Store the minimum once and the distances from it, and the values shrink to fit the spread rather than the magnitude. LIT verified live. 100,000 timestamps around 1,700,000,000 spanning 999 . The deltas need 10 bits each rather than 32 — 125,004 bytes against 400,000 , a factor of 3.2 . Every value reconstructs exactly: 0 errors across all 100,000, because the transform is subtraction and nothing is approximated. 2 HOW IT WAS WEAVED · AI + HUMAN Frame-of-reference encoding is standard in column stores and time-series databases, usually stacked with bit-packing on top of it. AVAN (AI) verified exact reconstruction rather than only measuring the ratio, because a compression figure means nothing without it. The number that carries the idea is 10 bits : the data was never 32 bits wide, it was 10 bits of information wearing a 32-bit costume, and the encoding does not compress anything — it stops storing a constant a hundred thousand times. 3 ONE DIMENSION The values, and the same values minus their minimum. 4 TWO DIMENSIONS · INTERACTIVE Widen the spread and watch the saving disappear. wider spread ▶ narrower reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: one offset, and a hundred thousand small numbers. AVAN’s addition (the inverse-companion): the forward reading is that frame-of-reference compresses a column dramatically. The inverse is that it compresses nothing and merely stops repeating yourself . The 22 high bits were identical in every row; they were never data, they were a fact about the column stored once per value. Read backwards, most impressive compression ratios are measurements of how much redundancy the format introduced in the first place, and the honest figure is not 3.2× but 10 bits — the amount that was ever there. pause spin LIT 100,000 timestamps around 1,700,000,000 spanning 999 need 10 bits per delta rather than 32 - 125,004 bytes against 400,000, a factor of 3.2 - and every value reconstructs exactly with 0 errors across all 100,000, because the transform is subtraction and nothing is approximated FIG Frame-of-reference encoding is standard in column stores and time-series databases, usually stacked with bit-packing on top. AVAN verified exact reconstruction rather than only measuring the ratio, because a compression figure means nothing without it. The number that carries the idea is 10 bits: the data was never 32 bits wide, it was 10 bits of information wearing a 32-bit costume, and the encoding stops storing a constant a hundred thousand times. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE INVENTORY · David Lee Wise (ROOT0), with AVAN


THE BITPACKING

THE BITPACKING · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · CHEAT · THE SPEEDRUN ◆ .dlw.fold THE FOLD / CHEAT / THE SPEEDRUN / THE BITPACKING THE BITPACKING the byte boundary was never waste, it was an index 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION If every value fits in five bits, storing each in a byte wastes three bits per value. Bit-packing ignores byte boundaries entirely and lays the values end to end. LIT verified live. 20,000 five-bit values pack into 12,500 bytes — exactly ceil(20000×5/8) , not approximately — against 20,000 at a byte each and 80,000 raw. That is 1.6× against byte-aligned and 6.4× against 32-bit. All 20,000 unpack to their original values with 0 errors, which is the part worth checking, because a packer that loses the last partial value would still produce a good ratio. 2 HOW IT WAS WEAVED · AI + HUMAN Bit-packing sits under every column store and inverted index, usually applied after frame-of-reference has made the values small. AVAN (AI) checked the round trip and the exact byte count together. The byte count matching ceil(N×W/8) proves nothing was silently padded per value; the 0 unpack errors prove nothing was dropped at the tail. Either check alone passes for an implementation that is quietly broken in the other direction. 3 ONE DIMENSION Five-bit values crossing byte boundaries. 4 TWO DIMENSIONS · INTERACTIVE Change the width and watch the packing change shape. wider values ▶ narrower reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: values that ignore the byte. AVAN’s addition (the inverse-companion): the forward reading is that bit-packing removes the padding waste. The inverse is that the byte boundary was never waste — it was an index . Byte-aligned data can be addressed, sliced, memory-mapped and read by anything; packed data must be decoded from a known start before any single value can be found. Read backwards, the three wasted bits per value were paying for random access, and the 1.6× is the price of that access rather than a free saving. pause spin LIT 20,000 five-bit values pack into 12,500 bytes - exactly ceil(20000 x 5/8), not approximately - against 20,000 at a byte each and 80,000 raw, giving 1.6x against byte-aligned and 6.4x against 32-bit, and all 20,000 unpack to their original values with 0 errors FIG Bit-packing sits under every column store and inverted index, usually applied after frame-of-reference has made the values small. AVAN checked the round trip and the exact byte count together: the byte count matching ceil(N x W/8) proves nothing was silently padded per value, and the 0 unpack errors prove nothing was dropped at the tail. Either check alone passes for an implementation quietly broken in the other direction. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE SPEEDRUN · David Lee Wise (ROOT0), with AVAN


THE ARITHMETIC CODER

Huffman gives every symbol a whole number of bits. If a symbol deserves 2.2 bits it gets 2 or 3, and the rounding is paid on every occurrence. An arithmetic coder narrows one interval instead.

THE ARITHMETIC CODER · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · LOOT · THE MINT ◆ .dlw.fold THE FOLD / LOOT / THE MINT / THE ARITHMETIC CODER THE ARITHMETIC CODER cheaper because it delivers something that is not a code 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Huffman gives every symbol a whole number of bits. If a symbol deserves 2.2 bits it gets 2 or 3, and the rounding is paid on every occurrence. An arithmetic coder does not assign codes to symbols at all — it narrows one interval. LIT verified live. 20,000 symbols over an 8 -letter alphabet with an entropy of 2.2027 bits. The ideal is 44,054 bits. Huffman spends 44,901 — 847 bits more, 1.92% over — and it cannot do better, because the overhead is the rounding and the rounding is structural. 2 HOW IT WAS WEAVED · AI + HUMAN Huffman ’s 1952 code is optimal among codes that assign whole bits to symbols; arithmetic coding, from Rissanen and Pasco in the 1970s, escapes by refusing that constraint. AVAN (AI) built the Huffman tree and counted its actual bits rather than quoting the bound. 1.92% is small, which is the honest finding — Huffman is very good. The gap matters at skewed alphabets where one symbol deserves a fraction of a bit and Huffman must still hand it a whole one. 3 ONE DIMENSION Ideal bits, and what whole-bit codes cost. 4 TWO DIMENSIONS · INTERACTIVE Skew the alphabet and watch the rounding bite. more skewed ▶ flatter reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: one interval, narrowing. AVAN’s addition (the inverse-companion): the forward reading is that arithmetic coding beats Huffman by escaping whole bits. The inverse is that it escapes them by never producing a code for anything . There is no codeword for a symbol, no table to look up, no place to start decoding in the middle — the output is one number and every symbol is smeared across all of it. Read backwards, Huffman’s 1.92% is the price of a code you can index, and arithmetic coding is cheaper because it delivers something that is not a code at all. pause spin LIT 20,000 symbols over an 8-letter alphabet with an entropy of 2.2027 bits give an ideal of 44,054 bits, where Huffman spends 44,901 - 847 bits more, 1.92% over - and it cannot do better because the overhead is the rounding and the rounding is structural FIG Huffman's 1952 code is optimal among codes that assign whole bits to symbols; arithmetic coding, from Rissanen and Pasco in the 1970s, escapes by refusing that constraint. AVAN built the Huffman tree and counted its actual bits rather than quoting the bound. 1.92% is small, which is the honest finding - Huffman is very good. The gap matters at skewed alphabets where one symbol deserves a fraction of a bit and Huffman must still hand it a whole one. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE MINT · David Lee Wise (ROOT0), with AVAN


THE BWT

Sort every rotation of a string and take the last column. Characters preceding the same context end up adjacent, so the result is full of runs - and it is exactly reversible from one integer.

THE BWT · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · CHEAT · NOCLIP ◆ .dlw.fold THE FOLD / CHEAT / NOCLIP / THE BWT THE BWT rearrangement so that compression becomes possible 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Sort every rotation of a string and take the last column. Characters that precede the same context end up next to each other, so the result is full of runs — and the whole thing is exactly reversible from one integer. LIT verified live. 2,000 characters of repeated text, 27 distinct symbols. Before: 2,000 runs, mean run length 1 . After: 40 runs, mean length 50 — a 50× reduction. It inverts back to the original exactly, and the entropy is unchanged to six decimal places: 4.3394 bits either way. 2 HOW IT WAS WEAVED · AI + HUMAN Burrows and Wheeler published this in 1994; it is the front end of bzip2 and the basis of the FM-index. AVAN (AI) got it wrong twice, and both corrections improved it. The inverse transform was simply broken — the reconstruction is the LF mapping, walked backwards from the stored row index. And it was first run on an i.i.d. source, where BWT should fail: it clusters characters by the context that follows them, and a memoryless source has no context. Runs went up , which was the right answer to a badly posed question. 3 ONE DIMENSION The string, and the last column of its sorted rotations. 4 TWO DIMENSIONS · INTERACTIVE Take away the structure and watch it stop working. structured text ▶ random source invert it 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: the same letters, in a better order. AVAN’s addition (the inverse-companion): the forward reading is that the Burrows–Wheeler transform prepares data for compression. The inverse is that it compresses nothing whatsoever . The output is a permutation — the same multiset of characters, the same entropy to six decimals, not one bit smaller. What it does is move the redundancy from a place no coder can see, spread across contexts, to a place every coder can, adjacency. Read backwards, this is not compression but rearrangement so that compression becomes possible , and the distinction is the whole idea. pause spin LIT 2,000 characters of repeated text over 27 distinct symbols go from 2,000 runs with a mean run length of 1 to 40 runs with a mean of 50 - a 50x reduction - inverting back to the original exactly, with the entropy unchanged to six decimal places at 4.3394 bits either way FIG Burrows and Wheeler published this in 1994; it is the front end of bzip2 and the basis of the FM-index. AVAN got it wrong twice and both corrections improved it. The inverse transform was simply broken - the reconstruction is the LF mapping walked backwards from the stored row index. And it was first run on an i.i.d. source, where BWT should fail: it clusters characters by the context that follows them and a memoryless source has no context, so runs went up, which was the right answer to a badly posed question. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of NOCLIP · David Lee Wise (ROOT0), with AVAN


THE LZ77 WINDOW

LZ77 replaces a repeat with a reference backwards. It can only reference what is still inside its window, so a pattern repeating further apart than the window is invisible.

THE LZ77 WINDOW · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · GRIND · WARM CACHE ◆ .dlw.fold THE FOLD / GRIND / WARM CACHE / THE LZ77 WINDOW THE LZ77 WINDOW it finds recent repeats, not repeats 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION LZ77 replaces a repeat with a reference backwards: how far, and how long. It can only reference what is still inside its window, so a pattern that repeats further apart than the window is invisible to it. LIT verified live. A string with a period of exactly 1,000 characters. At a window of 64 it costs 3,240 tokens and matches 27.8% of the input. At 1,024 — just past the period — it costs 495 and matches 93.8% . Widening to 4,096 changes nothing: 495 again. The window must clear the period, and beyond that it buys nothing. 2 HOW IT WAS WEAVED · AI + HUMAN Lempel and Ziv ’s 1977 scheme underlies DEFLATE, zstd and every zip file you have opened. AVAN (AI) chose a known period so the threshold would be a prediction rather than an observation. The interesting pair is 1,024 and 4,096 giving the identical token count: window size is not a dial that trades memory for ratio smoothly, it is a threshold with a flat region on either side, and the only question is which side of the period you are on. 3 ONE DIMENSION Window size against tokens. The cliff sits at the period. 4 TWO DIMENSIONS · INTERACTIVE Move the window across the period. wider window ▶ narrower reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a reference reaching backwards. AVAN’s addition (the inverse-companion): the forward reading is that a bigger window compresses better. The inverse is that the window is a statement about what you expect to matter, made before you look . Everything older is not merely uncompressed — it is unreachable, and the coder cannot tell the difference between data with no redundancy and redundancy it has forgotten. Read backwards, LZ77 does not find repeats; it finds recent repeats, and the ratio you get is a measurement of how well your guess about recency matched the file. pause spin LIT a string with a period of exactly 1,000 characters costs 3,240 tokens at a window of 64 while matching 27.8% of the input, and 495 tokens matching 93.8% at a window of 1,024 - just past the period - with 4,096 giving the identical 495, so the window must clear the period and beyond that buys nothing FIG Lempel and Ziv's 1977 scheme underlies DEFLATE, zstd and every zip file you have opened. AVAN chose a known period so the threshold would be a prediction rather than an observation. The interesting pair is 1,024 and 4,096 giving the identical token count: window size is not a dial trading memory for ratio smoothly, it is a threshold with a flat region on either side. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of WARM CACHE · David Lee Wise (ROOT0), with AVAN


THE DICTIONARY CODER

There are two different redundancies in a file: some symbols are more common than others, and some sequences repeat. An entropy coder sees only the first. A dictionary coder sees only the second.

THE DICTIONARY CODER · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · LOOT · THE INVENTORY ◆ .dlw.fold THE FOLD / LOOT / THE INVENTORY / THE DICTIONARY CODER THE DICTIONARY CODER incompressible is never a property of the data 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION There are two entirely different kinds of redundancy in a file: some symbols are more common than others, and some sequences repeat. An entropy coder sees only the first. A dictionary coder sees only the second. LIT verified live. 64 uniformly random characters, repeated 200 times — 12,800 characters. Its order-0 entropy is 4.4086 bits per symbol, so an entropy coder reports 56,430 bits and declares the data nearly incompressible. A dictionary stores the unit once and a count: 520 bits. Same string, 109× apart, and both coders are working correctly. 2 HOW IT WAS WEAVED · AI + HUMAN This is why DEFLATE is LZ77 followed by Huffman rather than either alone, and why compressing an already-compressed file achieves nothing. AVAN (AI) built a string designed so the two measures disagree maximally, because the point is not that dictionaries are better. On a file with skewed symbol frequencies and no repeats the answer inverts exactly. 109× is not a ranking — it is the size of the blind spot each coder has, measured on a case chosen to expose one of them. 3 ONE DIMENSION One string, two verdicts. 4 TWO DIMENSIONS · INTERACTIVE Trade repetition against skew and watch the winner swap. more repetition ▶ more skew reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: two blind spots, facing away. AVAN’s addition (the inverse-companion): the forward reading is that you need both kinds of coder. The inverse is that “incompressible” is never a property of the data . It is a report from one particular model that found nothing it was built to look for, and the same bytes are 109× smaller to a model with a different appetite. Read backwards, every compression ratio is a statement about the compressor, and a file is only random with respect to whoever is looking at it. pause spin LIT 64 uniformly random characters repeated 200 times - 12,800 characters - have an order-0 entropy of 4.4086 bits per symbol, so an entropy coder reports 56,430 bits and declares the data nearly incompressible, while a dictionary storing the unit once and a count needs 520 bits: the same string, 109 times apart, with both coders working correctly FIG This is why DEFLATE is LZ77 followed by Huffman rather than either alone, and why compressing an already-compressed file achieves nothing. AVAN built a string designed so the two measures disagree maximally, because the point is not that dictionaries are better - on a file with skewed frequencies and no repeats the answer inverts exactly. 109x is not a ranking, it is the size of the blind spot each coder has. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE INVENTORY · David Lee Wise (ROOT0), with AVAN


THE KOLMOGOROV BOUND

Most strings cannot be compressed at all, and this is not an empirical observation about real files - it is a counting argument, and it is airtight.

THE KOLMOGOROV BOUND · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · BOSS · THE WALL ◆ .dlw.fold THE FOLD / BOSS / THE WALL / THE KOLMOGOROV BOUND THE KOLMOGOROV BOUND real data lives in a corner the theorem is not about 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Most strings cannot be compressed at all, and this is not an empirical observation about real files — it is a counting argument, and it is airtight. LIT verified live by counting, not by experiment. Outputs of length at most n−k number 2 n-k+1 −1 , so the fraction that can be shortened by k bits is about 2 1-k . At every width tested, more than 50% of strings cannot be shortened by two bits. At 20 bits, 99.80% cannot be shortened by ten. Any compressor that shrinks your file did so by growing somebody else’s. 2 HOW IT WAS WEAVED · AI + HUMAN This is the counting form of the incompressibility theorem; Kolmogorov complexity is the general statement, and it is uncomputable, which the counting argument is not. AVAN (AI) counted rather than sampled deliberately. A measured claim about real files would be an observation about the files; this is a fact about the pigeonhole and holds for every compressor that has been written or ever will be. The percentages are arithmetic, and there is nothing to disagree with. 3 ONE DIMENSION Strings, and the shorter slots available to them. 4 TWO DIMENSIONS · INTERACTIVE Ask for more savings and watch the fraction collapse. save more bits ▶ save fewer wider strings reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: more pigeons than holes. AVAN’s addition (the inverse-companion): the forward reading is that almost nothing is compressible. The inverse is that almost nothing is data . The overwhelming majority of bit strings are not files anybody has, wants, or will ever produce — they are the noise the counting argument is about, and real data lives in a vanishingly small corner where structure is the rule. Read backwards, compression works spectacularly in practice precisely because the theorem is about a space we almost never visit, and every working compressor is a bet on which corner you live in. pause spin LIT outputs of length at most n-k number 2^(n-k+1) - 1, so the fraction that can be shortened by k bits is about 2^(1-k): at every width tested more than 50% of strings cannot be shortened by two bits, and at 20 bits 99.80% cannot be shortened by ten - any compressor that shrinks your file did so by growing somebody else's FIG This is the counting form of the incompressibility theorem; Kolmogorov complexity is the general statement and is uncomputable, which the counting argument is not. AVAN counted rather than sampled deliberately: a measured claim about real files would be an observation about the files, while this is a fact about the pigeonhole and holds for every compressor that has been written or ever will be. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE WALL · David Lee Wise (ROOT0), with AVAN


THE PIGEONHOLE COMPRESSION

A lossless compressor is an injective map: distinct inputs must give distinct outputs, or you cannot get the original back. That single requirement forbids shrinking everything.

THE PIGEONHOLE COMPRESSION · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · BOSS · THE GAUNTLET ◆ .dlw.fold THE FOLD / BOSS / THE GAUNTLET / THE PIGEONHOLE COMPRESSION THE PIGEONHOLE COMPRESSION compression was always an opinion 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION A lossless compressor is an injective map: distinct inputs must give distinct outputs, or you cannot get the original back. That single requirement forbids a compressor that shrinks everything. LIT verified live by counting. There are 65,536 strings of 16 bits and only 65,535 distinct outputs shorter than 16 bits — every length from 0 to 15 combined. So at least 1 input cannot shrink, and the same holds at every width tested: 2 , 4 , 6 , 8 , 10 , 12 , 14 , 16 . It is arithmetic, not a limitation of anybody’s algorithm. 2 HOW IT WAS WEAVED · AI + HUMAN This is the counting argument behind every “infinite compression” patent being rejected without reading the method. AVAN (AI) made the slot count explicit rather than stating the theorem. 65,535 against 65,536 is a difference of one, and one is enough — the argument does not need most strings to be incompressible, only that the destination is smaller than the source. Everything else about the compressor is irrelevant. 3 ONE DIMENSION Inputs against shorter slots, at every width. 4 TWO DIMENSIONS · INTERACTIVE Try to fit them in. Watch one always be left over. wider ▶ narrower reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: one pigeon with nowhere to go. AVAN’s addition (the inverse-companion): the forward reading is that no compressor can shrink every input. The inverse is that every compressor is therefore a declaration of taste . Since some inputs must grow, the designer chooses which ones — and that choice is the entire product. A compressor is not a machine that finds redundancy; it is a ranking of which files deserve to be small, expressed in code. Read backwards, the theorem does not limit compression, it reveals that compression was always an opinion. pause spin LIT there are 65,536 strings of 16 bits and only 65,535 distinct outputs shorter than 16 bits - every length from 0 to 15 combined - so at least 1 input cannot shrink, and the same holds at every width tested from 2 to 16: it is arithmetic, not a limitation of anybody's algorithm FIG This is the counting argument behind every infinite-compression patent being rejected without reading the method. AVAN made the slot count explicit rather than stating the theorem. 65,535 against 65,536 is a difference of one, and one is enough - the argument does not need most strings to be incompressible, only that the destination is smaller than the source. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE GAUNTLET · David Lee Wise (ROOT0), with AVAN


THE DELTA OF DELTA

Timestamps at a regular interval have constant first differences, so their second differences are almost all zero. Store those and the common case costs nothing.

THE DELTA OF DELTA · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · GRIND · THE EPOCH ◆ .dlw.fold THE FOLD / GRIND / THE EPOCH / THE DELTA OF DELTA THE DELTA OF DELTA an operational health metric wearing a storage costume 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Timestamps arriving at a regular interval have constant first differences, so their second differences are almost all zero. Store those, and the common case costs nothing at all. LIT verified live. 100,000 timestamps at a nominal ten-second cadence with occasional jitter. 93.42% of second differences are exactly 0 . The deltas need 5 bits; the delta-of-deltas need 4 — against 32 raw, an 8× reduction. Every value reconstructs exactly: 0 errors across all 100,000, because the transform is subtraction twice and nothing is approximated. 2 HOW IT WAS WEAVED · AI + HUMAN Delta-of-delta is the timestamp half of Facebook ’s Gorilla paper (2015) and is now standard in time-series stores. AVAN (AI) reports the 93.42% rather than the ratio, because the ratio is a consequence and the zero-fraction is the mechanism. A coder that spends a single bit on “same as last time” converts a regular cadence into almost nothing — and the moment the cadence stops being regular, the whole advantage disappears with it. 3 ONE DIMENSION Values, deltas, and delta-of-deltas. 4 TWO DIMENSIONS · INTERACTIVE Add jitter and watch the zeros disappear. more jitter ▶ less reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a difference of a difference. AVAN’s addition (the inverse-companion): the forward reading is that delta-of-delta exploits regular cadence. The inverse is that it is a bet on a machine behaving well, stored in your data format . The 93.42% is a measurement of how reliable somebody’s scheduler was, and a garbage collection pause or a clock correction converts your cheapest column into your most expensive one. Read backwards, the compression ratio of a time series is an operational health metric wearing a storage costume. pause spin LIT 100,000 timestamps at a nominal ten-second cadence with occasional jitter give 93.42% of second differences exactly 0, needing 4 bits where the deltas need 5 and the raw values need 32 - an 8x reduction - with every value reconstructing exactly and 0 errors across all 100,000 FIG Delta-of-delta is the timestamp half of Facebook's Gorilla paper (2015) and is now standard in time-series stores. AVAN reports the 93.42% rather than the ratio, because the ratio is a consequence and the zero-fraction is the mechanism. A coder spending a single bit on same-as-last-time converts a regular cadence into almost nothing - and the moment the cadence stops being regular the whole advantage goes with it. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE EPOCH · David Lee Wise (ROOT0), with AVAN


THE ENTROPY FLOOR

Shannon's entropy is not a target that good coders approach. It is a floor no coder can go under, and every real scheme sits some measurable distance above it.

THE ENTROPY FLOOR · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · BOSS · THE CHOKE POINT ◆ .dlw.fold THE FOLD / BOSS / THE CHOKE POINT / THE ENTROPY FLOOR THE ENTROPY FLOOR a property of your model, not of the data 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Shannon’s entropy is not a target that good coders approach. It is a floor that no coder can go under, and every real scheme sits some measurable distance above it. LIT verified live on one corpus. 30,000 symbols, 8 letters, entropy 2.229 bits per symbol — a floor of 66,870 bits. Fixed-width coding spends 90,000 : 34.59% above. Huffman spends 68,048 : 1.76% above. Both are above and neither is below, which is the whole content of the theorem, measured rather than asserted. 2 HOW IT WAS WEAVED · AI + HUMAN Shannon ’s source coding theorem (1948) sets the bound; everything since is engineering to approach it. AVAN (AI) ran two real coders against the same corpus rather than quoting the bound alone. A floor nobody tests is a claim; a floor two independent schemes sit above, at 34.59% and 1.76% , is a measurement. The gap between those two is also the finding — the distance from naive to near-optimal is twenty times the distance from near-optimal to perfect. 3 ONE DIMENSION The floor, and two coders above it. 4 TWO DIMENSIONS · INTERACTIVE Change the source and watch the floor move with it. more skewed ▶ flatter reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a floor nothing gets under. AVAN’s addition (the inverse-companion): the forward reading is that entropy bounds how far you can compress. The inverse is that the floor is a property of your model, not of the data . 2.229 bits is the entropy under an order-0 model that assumes symbols are independent; adopt a model with context and the same file has a lower floor, and the “bound” you could not cross moves. Read backwards, Shannon’s theorem does not say how small a file can be — it says how small it can be given what you have agreed to believe about it . pause spin LIT 30,000 symbols over 8 letters with an entropy of 2.229 bits per symbol give a floor of 66,870 bits, where fixed-width coding spends 90,000 at 34.59% above and Huffman spends 68,048 at 1.76% above - both above and neither below, which is the whole content of the theorem measured rather than asserted FIG Shannon's source coding theorem (1948) sets the bound; everything since is engineering to approach it. AVAN ran two real coders against the same corpus rather than quoting the bound alone. A floor nobody tests is a claim; a floor two independent schemes sit above, at 34.59% and 1.76%, is a measurement. The gap between those two is also the finding - the distance from naive to near-optimal is twenty times the distance from near-optimal to perfect. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE CHOKE POINT · David Lee Wise (ROOT0), with AVAN


THE RUN LENGTH

Replace each run of identical symbols with the symbol and a count. The simplest compression there is, and the clearest demonstration that no compressor helps everything.

THE RUN LENGTH · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · GLITCH · OFF BY ONE ◆ .dlw.fold THE FOLD / GLITCH / OFF BY ONE / THE RUN LENGTH THE RUN LENGTH it loses loudly where others lose quietly 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Replace each run of identical symbols with the symbol and a count. It is the simplest compression there is, and it is the clearest demonstration that no compressor helps everything. LIT verified live, same coder and same alphabet in all three columns. On run-structured input, 2,000 bytes become 80 — 25× smaller. On strictly alternating input they become 4,000 — exactly double , the worst case, because every run has length one and costs two bytes. On a random source, 3,496 . Two of the three columns are expansions. 2 HOW IT WAS WEAVED · AI + HUMAN Run-length coding is in fax machines, BMP files and the back end of BWT-based compressors, and it is the pigeonhole theorem you can see in one line. AVAN (AI) checked that the worst case is exactly doubling rather than merely bad, because 4,000 from 2,000 is a prediction and “expands” is not. It is the cleanest illustration available of the counting argument two spheres over: the coder that wins hardest also loses hardest, on the same alphabet, with nothing changed but the order. 3 ONE DIMENSION Three inputs, one coder. 4 TWO DIMENSIONS · INTERACTIVE Shorten the runs until the coder turns against you. shorter runs ▶ longer reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a count standing in for a run. AVAN’s addition (the inverse-companion): the forward reading is that run-length coding is naive and fails on unstructured data. The inverse is that its failure is the honest one . It doubles, loudly and predictably, on exactly the inputs it cannot help — where a sophisticated coder fails quietly by a few percent and leaves you believing it worked. Read backwards, the crudeness is a feature of the diagnostic: a compressor whose worst case is visible has told you something, and one whose worst case is invisible has only hidden it. pause spin LIT the same coder on the same alphabet turns 2,000 bytes of run-structured input into 80 - 25 times smaller - turns strictly alternating input into exactly 4,000, the worst case of doubling because every run has length one and costs two bytes, and turns a random source into 3,496: two of the three columns are expansions FIG Run-length coding is in fax machines, BMP files and the back end of BWT-based compressors, and it is the pigeonhole theorem you can see in one line. AVAN checked that the worst case is exactly doubling rather than merely bad, because 4,000 from 2,000 is a prediction and expands is not. The coder that wins hardest also loses hardest, on the same alphabet, with nothing changed but the order. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of OFF BY ONE · David Lee Wise (ROOT0), with AVAN


THE GOLOMB RICE

Golomb-Rice spends a value in two parts: the high bits in unary, the low k in binary. Choose k to match the distribution and it is near-optimal. Choose it badly and it is catastrophic.

THE GOLOMB RICE · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · CHEAT · THE SHORTCUT ◆ .dlw.fold THE FOLD / CHEAT / THE SHORTCUT / THE GOLOMB RICE THE GOLOMB RICE a tuned coder is a prediction nobody checks again 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Golomb–Rice coding spends a value in two parts: the high bits in unary, the low k in binary. Choose k to match the distribution and it is near-optimal. Choose it badly and it is catastrophic. LIT verified live. 20,000 geometric values with a mean of 16 , swept across every k from 0 to 8 . The best is k=3 at 5.515 bits per value; theory predicts log₂(mean) = 4 , and the measured optimum is within one. At k=0 the same data costs 16.297 bits — a 2.96× penalty for one wrong parameter on identical input. 2 HOW IT WAS WEAVED · AI + HUMAN Golomb ’s 1966 code is optimal for geometric sources; Rice ’s power-of-two variant is what FLAC and lossless image formats actually use. AVAN (AI) swept every parameter and compared the winner to the prediction, rather than coding at the predicted value and reporting that it worked. Those are different claims: the second assumes the theory, the first tests it. The measured optimum landing within one of log₂(mean) is the result, and the 2.95× penalty is what the theory is worth. 3 ONE DIMENSION Every k, and the cost of each. 4 TWO DIMENSIONS · INTERACTIVE Pick a k, or change the source underneath it. next k ▶ best k change the mean reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: unary on top, binary underneath. AVAN’s addition (the inverse-companion): the forward reading is that Golomb–Rice is near-optimal when tuned. The inverse is that the tuning is a claim about data you have not seen yet . k is fixed when the format is written and the source is free to change afterwards — so the 2.95× is not a penalty for incompetence but the standing risk of every parameter baked into a codec. Read backwards, a tuned coder is a prediction, and its compression ratio is the score on a forecast nobody checks again. pause spin LIT 20,000 geometric values with a mean of 16 swept across every k from 0 to 8 give a best of k=3 at 5.515 bits per value against a theoretical log2(mean) of 4 - the measured optimum within one of the prediction - while k=0 costs 16.297 bits on the same data, a 2.96x penalty for one wrong parameter FIG Golomb's 1966 code is optimal for geometric sources; Rice's power-of-two variant is what FLAC and lossless image formats actually use. AVAN swept every parameter and compared the winner to the prediction, rather than coding at the predicted value and reporting that it worked. Those are different claims: the second assumes the theory, the first tests it. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE SHORTCUT · David Lee Wise (ROOT0), with AVAN


THE PARTIAL FAILURE

A remote call has three outcomes, not two. It worked, it did not happen, or it happened and the answer was lost. From the caller the last two look identical.

THE PARTIAL FAILURE · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · GLITCH · HEISENBUG ◆ .dlw.fold THE FOLD / GLITCH / HEISENBUG / THE PARTIAL FAILURE THE PARTIAL FAILURE it makes the unknown harmless, not known 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION A remote call has three outcomes, not two. It worked, it did not happen, or — the one nobody designs for — it happened and the answer was lost. From the caller those last two look identical. LIT verified live. 100,000 calls: 94,039 succeeded, 2,965 cleanly failed before arriving, and 2,996 arrived and lost their reply — 3.00% in a state the caller cannot distinguish. Only the 2,965 are safe to retry blindly. Retrying the other 2,996 duplicates real work. With an idempotency key all 100,000 become safe, because the question changes from “did it happen” to “has this one happened”. 2 HOW IT WAS WEAVED · AI + HUMAN The three-outcome problem is why every payments API has an idempotency key and why “at least once” is the only delivery guarantee most systems can actually offer. AVAN (AI) counted the buckets separately rather than reporting a success rate. 94% succeeded is the number that gets published; 3% unknown is the number that decides your architecture, and averaging them into “97% did not fail” hides exactly the population that needs the design work. 3 ONE DIMENSION Three outcomes, and which two the caller can tell apart. 4 TWO DIMENSIONS · INTERACTIVE Retry, and watch the duplicates arrive. retry blindly ▶ add idempotency key reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a question with no answer coming. AVAN’s addition (the inverse-companion): the forward reading is that partial failure is a hard case to handle. The inverse is that it is the only case, and the other two are conveniences . Success and clean failure are both just partial failure where the evidence happened to survive; nothing about the network promised you that evidence. Read backwards, an idempotency key does not solve the unknown — it makes the unknown harmless, which is the only kind of solution available when the fact you need is genuinely not in your possession. pause spin LIT 100,000 calls give 94,039 successes, 2,965 clean failures that never arrived, and 2,996 that arrived and lost their reply - 3.00% in a state the caller cannot distinguish - so only the 2,965 are safe to retry blindly while retrying the other 2,996 duplicates real work, and an idempotency key makes all 100,000 safe FIG The three-outcome problem is why every payments API has an idempotency key and why at-least-once is the only delivery guarantee most systems can offer. AVAN counted the buckets separately rather than reporting a success rate: 94% succeeded is the number that gets published, 3% unknown is the number that decides your architecture, and averaging them into 97% did not fail hides exactly the population that needs the design work. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of HEISENBUG · David Lee Wise (ROOT0), with AVAN


THE POISON MESSAGE

THE POISON MESSAGE · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · GLITCH · RACE CONDITION ◆ .dlw.fold THE FOLD / GLITCH / RACE CONDITION / THE POISON MESSAGE THE POISON MESSAGE busy, at full CPU, making no progress 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION A queue with ordered delivery and infinite retry has a failure mode with no moving parts: one message that can never be processed stops every message behind it, forever. LIT verified live. 1,000 messages, one poison at position 7 . With unlimited retry the consumer processes 7 messages, burns 200,000 attempts, and never drains. With a dead-letter queue after 3 attempts it processes 999 , spends 1,002 attempts, sets 1 message aside, and finishes. One message held 992 others hostage. 2 HOW IT WAS WEAVED · AI + HUMAN Dead-letter queues exist for exactly this; the failure is common enough that ordered-delivery systems treat a retry cap as mandatory rather than optional. AVAN (AI) reports the work column alongside the throughput one, because 200,000 attempts for 7 messages is the part that hurts. The consumer is not idle or crashed — it is fully busy, at maximum CPU, making no progress, which is the hardest failure to spot on a dashboard. 3 ONE DIMENSION The queue, with and without a retry cap. 4 TWO DIMENSIONS · INTERACTIVE Set the retry cap and watch the queue drain or not. raise the cap ▶ lower unlimited reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a queue stopped at position seven. AVAN’s addition (the inverse-companion): the forward reading is that a dead-letter queue rescues the consumer. The inverse is that it rescues it by abandoning correctness for one message . The retry cap is a decision that after three tries you will stop trying to do something you were asked to do, and set it aside where nobody is watching. Read backwards, ordered delivery and guaranteed processing cannot both survive a message that will never succeed, and every dead-letter queue is a quiet admission of which one was given up. pause spin LIT 1,000 messages with one poison at position 7: unlimited retry processes 7, burns 200,000 attempts and never drains, while a dead-letter queue after 3 attempts processes 999, spends 1,002 attempts, sets 1 message aside and finishes - one message held 992 others hostage FIG Dead-letter queues exist for exactly this; the failure is common enough that ordered-delivery systems treat a retry cap as mandatory. AVAN reports the work column alongside the throughput one, because 200,000 attempts for 7 messages is the part that hurts. The consumer is not idle or crashed - it is fully busy, at maximum CPU, making no progress, which is the hardest failure to spot on a dashboard. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of RACE CONDITION · David Lee Wise (ROOT0), with AVAN


THE SPLIT BRAIN

Partition a cluster and both halves can decide they are in charge. Requiring a strict majority makes that arithmetically impossible - and the arithmetic also explains why clusters come in odd numbers.

THE SPLIT BRAIN · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · CO-OP · THE MERGE ◆ .dlw.fold THE FOLD / CO-OP / THE MERGE / THE SPLIT BRAIN THE SPLIT BRAIN it lets the smaller side disqualify itself 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Partition a cluster and both halves can decide they are in charge. Requiring a strict majority makes that arithmetically impossible — and the arithmetic also explains why clusters come in odd numbers. LIT verified live and exhaustively. Every partition of a 5 -node cluster — all 32 — enumerated. Naive election gives two leaders in 30 of them. Quorum gives two leaders in 0 , at every size from 3 to 8. But look at the deadlocks: odd sizes leave 0 partitions with no leader at all, while 4 nodes deadlock in 6 of 16 — 37.5% . Adding a fifth node to a four-node cluster does not add capacity; it removes the tie. 2 HOW IT WAS WEAVED · AI + HUMAN Quorum intersection is the guarantee, and its arithmetic is measured next door in THE PAXOS QUORUM — that sphere proves the safety side, that no two majority quorums are disjoint. This one measures what the safety costs: the odd-size convention is the folklore that follows from it. AVAN (AI) found the odd-size result by getting a gate wrong. I had asserted that some partition of five nodes would leave nobody in charge; it never does, because one side always holds three. Sweeping 3 to 8 instead of testing one size turned a failed assertion into the actual finding: even clusters deadlock on the tie and odd ones cannot. 3 ONE DIMENSION Cluster size against deadlocked partitions. 4 TWO DIMENSIONS · INTERACTIVE Cut the cluster and see who may lead. move the cut ▶ grow the cluster toggle quorum reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: one cluster, cut in two. AVAN’s addition (the inverse-companion): the forward reading is that quorum prevents split brain. The inverse is that it prevents it by choosing unavailability . The minority side is working perfectly, holds all its data and can reach its users — and it refuses to serve them, because it cannot prove it is not the one that was cut off. Read backwards, quorum does not detect the partition or resolve it; it hands the same rule to both halves and lets the smaller one disqualify itself, which is the only move available when neither side can see the other. pause spin LIT every partition of a 5-node cluster - all 32 - enumerated gives two leaders in 30 under naive election and 0 under quorum, at every size from 3 to 8; and the deadlocks are the finding, since odd sizes leave 0 partitions with no leader while 4 nodes deadlock in 6 of 16, which is 37.5% FIG Quorum intersection is the guarantee, and its arithmetic is measured next door in THE PAXOS QUORUM - that sphere proves the safety side, that no two majority quorums are disjoint, while this one measures what the safety costs. The odd-size convention is the folklore that follows. AVAN found the odd-size result by getting a gate wrong: I asserted that some partition of five nodes would leave nobody in charge, and it never does, because one side always holds three. Sweeping 3 to 8 instead of testing one size turned a failed assertion into the actual finding - even clusters deadlock on the tie and odd ones cannot. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE MERGE · David Lee Wise (ROOT0), with AVAN


THE GRAY FAILURE

The worst kind of broken is the kind that answers the health check. A node failing only on the path your users take stays in rotation because the thing watching it is not doing what they are doing.

THE GRAY FAILURE · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · BOSS · THE GATEKEEPER ◆ .dlw.fold THE FOLD / BOSS / THE GATEKEEPER / THE GRAY FAILURE THE GRAY FAILURE the shallowness is restraint, not laziness 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION The worst kind of broken is the kind that answers the health check. A node that is slow, or failing only on the path your users take, stays in rotation because the thing watching it is not doing what they are doing. LIT verified live. 200,000 samples of the same node. The health probe — small, cached, no dependencies — succeeds 99.91% of the time. Real requests, which touch the degraded dependency, succeed 61.93% . That is a gap of 37.99 points on one machine at one moment, and the node is not removed from rotation, because nothing that decides rotation ever saw the second number. 2 HOW IT WAS WEAVED · AI + HUMAN Gray failure was named by Huang et al. (HotOS 2017); the definition is precisely this differential observability between the system’s view and the user’s. AVAN (AI) measured both populations rather than describing the idea, because the number that matters is the gap , not either rate. 99.91% is a true statement about the probe. 61.93% is a true statement about the users. Nothing is lying, and the node stays up. 3 ONE DIMENSION What the probe sees, and what the users see. 4 TWO DIMENSIONS · INTERACTIVE Make the probe more like a real request. probe touches the dependency ▶ back to shallow reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: two observers, one machine. AVAN’s addition (the inverse-companion): the forward reading is that shallow health checks miss real failures. The inverse is that a health check deep enough to catch this becomes the outage . A probe that exercises every dependency fails whenever any dependency is briefly slow, and a fleet that removes nodes on that signal removes all of them at once. Read backwards, the shallowness is not laziness — it is the thing stopping the health system from being the largest source of downtime, and gray failure is the price of that restraint. pause spin LIT 200,000 samples of one node give a health probe - small, cached, no dependencies - succeeding 99.91% of the time while real requests touching the degraded dependency succeed 61.93%, a gap of 37.99 points on the same machine at the same moment, and the node is not removed from rotation because nothing that decides rotation saw the second number FIG Gray failure was named by Huang et al. (HotOS 2017); the definition is precisely this differential observability between the system's view and the user's. AVAN measured both populations rather than describing the idea, because the number that matters is the gap, not either rate. 99.91% is a true statement about the probe and 61.93% is a true statement about the users - nothing is lying, and the node stays up. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE GATEKEEPER · David Lee Wise (ROOT0), with AVAN


THE CASCADING FAILURE

A dependency loses capacity. Requests fail. Every failure is retried, so the offered load rises, so more fail. The retries are load, and the load is what caused the retries.

THE CASCADING FAILURE · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · BOSS · THE WALL ◆ .dlw.fold THE FOLD / BOSS / THE WALL / THE CASCADING FAILURE THE CASCADING FAILURE no faulty component anywhere in it 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION A dependency loses capacity. Requests fail. Every failure is retried, so the offered load rises, so more fail, so more are retried. The retries are load, and the load is what caused the retries. LIT verified live. Base load 1,000 , healthy capacity 1,200 , two retries per failure. Capacity drops to 600 for three rounds: offered load climbs to 3,400 with 2,800 failing. Capacity is then fully restored at round 5 — and by round 9 the offered load is 84,600 , an amplification of 84.60× , with 83,400 still failing. The trigger is gone and the failure is not. 2 HOW IT WAS WEAVED · AI + HUMAN Retry amplification is why every serious client library has a retry budget rather than a retry count, and why load shedding is applied at the caller. AVAN (AI) set the healthy capacity above the base load at first, so nothing ever failed and there was no cascade to measure — a clean run proving nothing. The version that means something needs the dependency to actually lose capacity. Restoring it at round 5 is the important half: 84,600 against a base of 1,000 , four rounds after the cause was removed. 3 ONE DIMENSION Offered load per round. The capacity comes back at round 5. 4 TWO DIMENSIONS · INTERACTIVE Change the retry policy and run it again. more retries ▶ fewer retry budget reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: load feeding on its own failure. AVAN’s addition (the inverse-companion): the forward reading is that retries cause cascading failure. The inverse is that every retry was individually correct . Each client saw a failed request and did the reasonable thing; not one of them misbehaved, and no single caller sent enough traffic to matter. Read backwards, this is a failure with no faulty component anywhere in it — the system is the defect, assembled entirely out of correct parts each doing what it was told. pause spin LIT base load 1,000 against a healthy capacity of 1,200 with two retries per failure: when capacity drops to 600 for three rounds the offered load climbs to 3,400 with 2,800 failing, and after capacity is fully restored at round 5 the offered load reaches 84,600 by round 9 - an amplification of 84.60x with 83,400 still failing, four rounds after the cause was removed FIG Retry amplification is why every serious client library has a retry budget rather than a retry count, and why load shedding is applied at the caller. AVAN set the healthy capacity above the base load at first, so nothing ever failed and there was no cascade to measure - a clean run proving nothing. Restoring capacity at round 5 is the important half: 84,600 against a base of 1,000, long after the trigger is gone. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE WALL · David Lee Wise (ROOT0), with AVAN


THE METASTABLE FAILURE

Some systems have two thresholds: the load at which they break, and the much lower load at which they recover. Between them they stay broken with no cause present.

THE METASTABLE FAILURE · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · RESPAWN · ROLLBACK ◆ .dlw.fold THE FOLD / RESPAWN / ROLLBACK / THE METASTABLE FAILURE THE METASTABLE FAILURE the only fix is refusing traffic you can serve 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Some systems have two thresholds: the load at which they break, and the much lower load at which they recover. Between them the system stays broken with no cause present, sustained entirely by its own reaction to being broken. LIT verified live. Capacity 1,000 , retry amplification 3× . It breaks at an offered load of 1,010 — just past capacity, as expected. It recovers only when offered load falls to 330 , because until then the retries alone exceed capacity. The gap is 680 requests, 68.0% of capacity: removing the trigger is not enough, and the load must drop to a third of what the system could originally serve. 2 HOW IT WAS WEAVED · AI + HUMAN Metastable failure was characterised by Bronson et al. (HotOS 2021); the defining feature is exactly this sustaining effect that outlives its trigger. AVAN (AI) computed both thresholds rather than describing hysteresis, because the gap is the whole phenomenon and it is a number. 1,010 to break, 330 to recover. An operator watching load return to normal sees a system that should be fine and is not, and the instinct — restart it, send the traffic back — puts it straight back over the line. 3 ONE DIMENSION Two thresholds, and the gap between them. 4 TWO DIMENSIONS · INTERACTIVE Push it over, then try to bring it back. more load ▶ less load stronger retries reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a state that holds itself down. AVAN’s addition (the inverse-companion): the forward reading is that metastable failure is a system stuck below its capacity. The inverse is that it is a system doing its job perfectly at the wrong equilibrium . Nothing is degraded; every component is serving exactly as much as it can, and the work arriving is genuine work that genuine clients want done. Read backwards, there is no fault to find and nothing to repair — the only intervention that works is to refuse traffic you are able to serve, which is why the fix always feels wrong to the person who has to do it. pause spin LIT with a capacity of 1,000 and a retry amplification of 3x the system breaks at an offered load of 1,010, just past capacity, but recovers only when offered load falls to 330, because until then the retries alone exceed capacity - a gap of 680 requests or 68.0% of capacity, so load must drop to a third of what the system could originally serve FIG Metastable failure was characterised by Bronson et al. (HotOS 2021); the defining feature is exactly this sustaining effect that outlives its trigger. AVAN computed both thresholds rather than describing hysteresis, because the gap is the whole phenomenon and it is a number. An operator watching load return to normal sees a system that should be fine and is not, and the instinct - restart it, send the traffic back - puts it straight back over the line. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of ROLLBACK · David Lee Wise (ROOT0), with AVAN


THE CORRELATED FAILURE

Three replicas at 99% give six nines - if they fail independently. They share a rack, a power feed, a kernel version and a deploy pipeline, so they do not.

THE CORRELATED FAILURE · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · BOSS · THE GAUNTLET ◆ .dlw.fold THE FOLD / BOSS / THE GAUNTLET / THE CORRELATED FAILURE THE CORRELATED FAILURE an availability figure is a belief about shared fate 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Three replicas at 99% give six nines — if they fail independently. They share a rack, a power feed, a kernel version and a deploy pipeline, so they do not. LIT verified live. With a 1% single-replica failure rate and perfect independence, all three fail together with probability 0.000001 : 6.00 nines. Fully correlated, they fail together with probability 0.01 — 2.00 nines, the same as one replica, and 10,000× worse. The collapse is not gradual: at a correlation of just 0.2 the figure is already 2.70 nines. Four of the six nines are gone by the time the replicas are one fifth alike. 2 HOW IT WAS WEAVED · AI + HUMAN Correlated failure is why availability targets are written per failure domain, and why “multi-AZ” means something specific rather than “three copies”. AVAN (AI) swept the correlation rather than contrasting the endpoints, because the endpoints suggest a trade-off and the sweep shows a cliff. Going from independent to 0.2 correlated costs more nines than going from 0.2 to fully correlated. The damage is done by the first small amount of shared fate. 3 ONE DIMENSION Correlation against nines. The cliff is at the left. 4 TWO DIMENSIONS · INTERACTIVE Add replicas, or admit they share something. more replicas ▶ more correlation reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: three copies of one fate. AVAN’s addition (the inverse-companion): the forward reading is that correlation destroys the benefit of replication. The inverse is that the independence was never measured, only assumed . Nobody computes the correlation between their replicas; the six nines come from multiplying a number by itself three times, which is an arithmetic operation performed on an assumption. Read backwards, the availability figure in the design document is not a prediction about the system — it is a restatement of a belief about how much the replicas have in common, and that belief is usually never written down at all. pause spin LIT a 1% single-replica failure rate with perfect independence gives all three failing together at probability 0.000001, which is 6.00 nines, while fully correlated they fail together at 0.01 - 2.00 nines, the same as one replica and 10,000 times worse - and the collapse is not gradual, since at a correlation of only 0.2 the figure is already 2.70 nines FIG Correlated failure is why availability targets are written per failure domain and why multi-AZ means something specific rather than three copies. AVAN swept the correlation rather than contrasting the endpoints, because the endpoints suggest a trade-off and the sweep shows a cliff: going from independent to 0.2 correlated costs more nines than going from 0.2 to fully correlated. The damage is done by the first small amount of shared fate. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE GAUNTLET · David Lee Wise (ROOT0), with AVAN


THE SILENT CORRUPTION

A disk does not always tell you when it returns the wrong bytes. Without a checksum there is no failure to detect: the read succeeds, the data is wrong, and nothing reports a problem.

THE SILENT CORRUPTION · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · LOOT · THE VAULT ◆ .dlw.fold THE FOLD / LOOT / THE VAULT / THE SILENT CORRUPTION THE SILENT CORRUPTION a visible outage chosen over an invisible corruption 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION A disk does not always tell you when it returns the wrong bytes. Without a checksum there is no failure to detect: the read succeeds, the data is wrong, and nothing anywhere reports a problem. LIT verified live. 200,000 four-byte blocks under a bit error rate of 1e-4 . 670 blocks were corrupted by a single flipped bit. A CRC-8 caught 670 of 670 — 100.00% , with an escape rate of 0 , because a single-bit error is exactly what a CRC is built to catch. Without the checksum all 670 pass silently, and the read reports success every time. 2 HOW IT WAS WEAVED · AI + HUMAN Silent data corruption is why ZFS and modern filesystems checksum every block, and why Bairavasundaram et al. found real corruption at rates that made it a design requirement rather than a curiosity. AVAN (AI) tested single-bit flips, which is the case a CRC is designed for — so 100% here is not a claim about CRC strength in general. A CRC-8 has a 1 in 256 escape rate against arbitrary multi-bit corruption. The honest finding is the comparison: 670 caught against 670 silent, with the only difference being whether anybody wrote the checksum down. 3 ONE DIMENSION Corrupted blocks, caught and silent. 4 TWO DIMENSIONS · INTERACTIVE Raise the error rate and watch what a read reports. worse media ▶ better remove the checksum reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a read that succeeded and was wrong. AVAN’s addition (the inverse-companion): the forward reading is that checksums detect silent corruption. The inverse is that they convert a data problem into an availability problem, on purpose . Before the checksum you had wrong bytes and a working system; after it you have a read that fails and an application that cannot proceed. Read backwards, the checksum does not save the data — the data was already gone — it forces someone to find out, and every layer that adds one is choosing a visible outage over an invisible corruption. pause spin LIT 200,000 four-byte blocks under a bit error rate of 1e-4 leave 670 corrupted by a single flipped bit, of which a CRC-8 catches 670 - 100.00% with an escape rate of 0, because a single-bit error is exactly what a CRC is built to catch - while without the checksum all 670 pass silently and the read reports success every time FIG Silent data corruption is why ZFS and modern filesystems checksum every block. AVAN tested single-bit flips, which is the case a CRC is designed for, so 100% here is not a claim about CRC strength in general - a CRC-8 has roughly a 1 in 256 escape rate against arbitrary multi-bit corruption. The honest finding is the comparison: 670 caught against 670 silent, with the only difference being whether anybody wrote the checksum down. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE VAULT · David Lee Wise (ROOT0), with AVAN


THE BACKPRESSURE

An unbounded queue never rejects anything, which sounds generous until you notice what it does instead: accepting work it will not get to for hours, and making everyone wait behind it.

THE BACKPRESSURE · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · BOSS · THE CHOKE POINT ◆ .dlw.fold THE FOLD / BOSS / THE CHOKE POINT / THE BACKPRESSURE THE BACKPRESSURE an invisible slow failure made a visible fast one 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION An unbounded queue never rejects anything, which sounds generous until you notice what it is doing instead: accepting work it will not get to for hours, and making everyone wait behind it. LIT verified live. 20,000 ticks, offered above capacity. Unbounded: 0 rejected, final queue 6,917 deep, mean wait 3,461.9 . Bounded at 100 : 6,818 shed, final queue 99 , mean wait 99.4 — 34.8× shorter. And the throughput is identical: 20,000 served either way, a difference of 0 . The unbounded queue served nobody extra; it only made the served ones wait. 2 HOW IT WAS WEAVED · AI + HUMAN This is Little’s Law with a policy attached: throughput is set by the server, and the queue only decides how long the wait is. AVAN (AI) put the throughput columns side by side because that is the whole argument and it is the column people expect to differ. 20,000 and 20,000 . Accepting the extra work bought exactly nothing, and the 6,818 requests that were shed would have waited an hour to be told the same thing. 3 ONE DIMENSION Two queues, one server, same arrivals. 4 TWO DIMENSIONS · INTERACTIVE Set the bound and watch the wait, not the throughput. deeper queue ▶ shallower unbounded reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a queue that says no. AVAN’s addition (the inverse-companion): the forward reading is that backpressure protects the system. The inverse is that it moves the failure to where somebody can see it . The unbounded queue fails too — it fails as a timeout, in a client, minutes later, with no error anywhere in your logs. Read backwards, shedding does not reduce the number of unhappy users by one; it changes an invisible slow failure into a visible fast one, and every argument against it is really an argument for not being the one who says no. pause spin LIT 20,000 ticks offered above capacity give an unbounded queue 0 rejected, a final depth of 6,917 and a mean wait of 3,461.9, against a queue bounded at 100 which sheds 6,818, ends 99 deep and waits 99.4 - 34.8 times shorter - while serving the identical 20,000, a throughput difference of 0 FIG This is Little's Law with a policy attached: throughput is set by the server, and the queue only decides how long the wait is. AVAN put the throughput columns side by side because that is the whole argument and it is the column people expect to differ. 20,000 and 20,000. Accepting the extra work bought exactly nothing, and the 6,818 requests that were shed would have waited an hour to be told the same thing. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE CHOKE POINT · David Lee Wise (ROOT0), with AVAN


THE CIRCUIT BREAKER

If a dependency has failed the last five times, the sixth call is not a request - it is a guess with a timeout attached. A circuit breaker stops guessing and checks back occasionally instead.

THE CIRCUIT BREAKER · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · RESPAWN · SECOND WIND ◆ .dlw.fold THE FOLD / RESPAWN / SECOND WIND / THE CIRCUIT BREAKER THE CIRCUIT BREAKER a dependency outage converted into your own, deliberately 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION If a dependency has failed the last five times, the sixth call is not a request — it is a guess with a timeout attached. A circuit breaker stops guessing and checks back occasionally instead. LIT verified live. A 2,000 -tick outage inside a 6,000 -tick run. Without a breaker: 6,000 calls, 2,000 failures — every single request during the outage waits for a timeout. With a breaker that opens after 5 consecutive failures and probes every 200 ticks: 14 failures and 1,990 rejected immediately. That is 1,986 wasted calls avoided, a 99.3% reduction, at the cost of 10 probe calls to notice the recovery. 2 HOW IT WAS WEAVED · AI + HUMAN The pattern is Michael Nygard ’s, from Release It! ; the half-open state is the part that makes it a breaker rather than a fuse. AVAN (AI) counted the probes as well as the savings, because the probe interval is the real design parameter. 10 probes across the outage is what buys the recovery detection, and shortening the interval to notice faster is precisely what turns the breaker back into the retry storm it was installed to stop. 3 ONE DIMENSION The run, with and without a breaker. 4 TWO DIMENSIONS · INTERACTIVE Tune the probe interval and the trip threshold. probe sooner ▶ probe later trip faster reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a door that closes itself. AVAN’s addition (the inverse-companion): the forward reading is that a breaker protects you from a failing dependency. The inverse is that it fails your requests on the dependency’s behalf, using stale evidence . Once open it rejects calls that might have succeeded, based on what was true two hundred ticks ago, and every rejection is a decision made without asking. Read backwards, a breaker converts a dependency’s outage into your own deliberate outage — faster, cheaper, and entirely your responsibility. pause spin LIT a 2,000-tick outage inside a 6,000-tick run costs 6,000 calls and 2,000 failures without a breaker, while a breaker opening after 5 consecutive failures and probing every 200 ticks gives 14 failures and 1,990 immediate rejections - 1,986 wasted calls avoided, a 99.3% reduction, at the cost of 10 probe calls to notice the recovery FIG The pattern is Michael Nygard's, from Release It!; the half-open state is what makes it a breaker rather than a fuse. AVAN counted the probes as well as the savings, because the probe interval is the real design parameter. 10 probes across the outage is what buys the recovery detection, and shortening the interval to notice faster is precisely what turns the breaker back into the retry storm it was installed to stop. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of SECOND WIND · David Lee Wise (ROOT0), with AVAN


THE CONVOY EFFECT

First-come-first-served is the fairest-sounding rule there is, and the one that makes almost everybody wait longest. A few long jobs at the front, and the whole short queue sits behind them.

THE CONVOY EFFECT · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · BOSS · THE CHOKE POINT ◆ .dlw.fold THE FOLD / BOSS / THE CHOKE POINT / THE CONVOY EFFECT THE CONVOY EFFECT the total wait is fixed; order decides whose it is 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION First-come-first-served is the fairest-sounding rule there is, and it is the one that makes almost everybody wait longest. A few long jobs at the front, and the whole short queue sits behind them. LIT verified live. 200 jobs, 12 long and 188 short. FIFO gives a mean wait of 1,540 . Shortest-job-first gives 409 — 3.77× shorter. The total work is identical: both finish at 3,689 . Nothing was made faster and nothing was dropped; the same jobs ran on the same machine for the same total time, and the only thing that changed was the order. 2 HOW IT WAS WEAVED · AI + HUMAN The convoy effect is the standard argument for SJF, and the standard argument against it is that it needs to know the job lengths — which is what THE MULTILEVEL FEEDBACK gets around. AVAN (AI) published the makespan next to the mean wait because that is the column that shows nothing was stolen. 3,689 either way. A scheduler cannot create throughput; it can only decide who does the waiting, and FIFO decides that 188 short jobs should wait for 12 long ones. 3 ONE DIMENSION The same jobs, two orders. 4 TWO DIMENSIONS · INTERACTIVE Reorder the queue and watch the wait, not the finish. sort shortest first ▶ longest first back to arrival order 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: a queue behind one long thing. AVAN’s addition (the inverse-companion): the forward reading is that SJF is better than FIFO. The inverse is that it is better for the mean and worse for the long job, and the mean has no opinion about that . Every one of those 188 short jobs gained, and the 12 long ones paid for all of it. Read backwards, “ 3.77× improvement” is a statement about a population, and there is no arrangement of a queue that improves it for everybody — the total wait is fixed by the job lengths, and a scheduler only ever decides whose it is. pause spin LIT 200 jobs - 12 long and 188 short - give a FIFO mean wait of 1,540 against 409 for shortest-job-first, which is 3.77 times shorter, while the total work is identical at a makespan of 3,689 either way, so nothing was made faster and nothing was dropped and the only thing that changed was the order FIG The convoy effect is the standard argument for SJF, and the standard argument against it is that it needs to know the job lengths - which is what THE MULTILEVEL FEEDBACK gets around. AVAN published the makespan next to the mean wait because that is the column showing nothing was stolen: 3,689 either way. A scheduler cannot create throughput, it can only decide who does the waiting, and FIFO decides that 188 short jobs should wait for 12 long ones. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE CHOKE POINT · David Lee Wise (ROOT0), with AVAN


THE WORK STEALING

THE WORK STEALING · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · CO-OP · THE MERGE ◆ .dlw.fold THE FOLD / CO-OP / THE MERGE / THE WORK STEALING THE WORK STEALING it does not schedule more cleverly, it schedules later 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Deciding who does what before you start only works if you already know how long each piece takes. When you do not, seven workers finish early and stand still while the eighth is buried. LIT verified live. 2,000 tasks, 74,359 units of work, 8 workers. A fixed block split finishes at 35,505 — 3.82× the ideal 9,294.9 , with the fleet idle 73.82% of the time. Letting a free worker take the next task off the most loaded queue finishes at 9,511 : 2.3% off ideal, 2.27% idle, 3.73× faster, after 358 steals. Same tasks, same workers, same total work. 2 HOW IT WAS WEAVED · AI + HUMAN Work stealing is the scheduler in Cilk, in Go’s runtime, in Java’s fork/join pool; the deque-with-stealing shape is Blumofe and Leiserson ’s. AVAN (AI) first ran this with tasks dealt round-robin, and measured 0 steals — because 250 random draws per worker converge to the same sum, so there was nothing to steal. That is not the case the technique exists for. Static partitioning only loses when the work is skewed , and the experiment has to put the skew somewhere a fixed split cannot see it. 3 ONE DIMENSION Eight workers, decided up front and decided as it goes. 4 TWO DIMENSIONS · INTERACTIVE Change the skew and watch the fixed split fail. more skew ▶ less skew toggle stealing reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: seven idle, one buried. AVAN’s addition (the inverse-companion): the forward reading is that stealing beats planning. The inverse is that the plan was never wrong — it was made too early . A fixed split is optimal for the information available when it is made, and it stays committed to that information after the work has told you something better. Read backwards, work stealing does not schedule more cleverly; it schedules later , and almost all of the 3.73× is the value of not having decided yet. pause spin LIT 2,000 tasks totalling 74,359 units across 8 workers finish at 35,505 under a fixed block split - 3.82 times the ideal 9,294.9, with the fleet idle 73.82% of the time - against 9,511 when a free worker takes the next task off the most loaded queue, which is 2.3% off ideal and 2.27% idle, a speedup of 3.73 after 358 steals FIG Work stealing is the scheduler in Cilk, in Go's runtime and in Java's fork/join pool; the deque-with-stealing shape is Blumofe and Leiserson's. AVAN first ran this with tasks dealt round-robin and measured 0 steals, because 250 random draws per worker converge to the same sum and there was nothing to steal. Static partitioning only loses when the work is skewed, and the experiment has to put the skew somewhere a fixed split cannot see it. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE MERGE · David Lee Wise (ROOT0), with AVAN


THE FAIR SHARE

Splitting a resource evenly is not fair when some claimants want less than their share. Max-min fairness gives the small ones everything they asked for first, then divides what is left.

THE FAIR SHARE · WORLD II — THE FOLD ◀ THE FOLD 0ROOT.AI // WORLD II · CO-OP · THE SYNC ◆ .dlw.fold THE FOLD / CO-OP / THE SYNC / THE FAIR SHARE THE FAIR SHARE an allocation built on self-reported need 1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION Splitting a resource evenly is not fair when some of the claimants want less than their share. Max-min fairness gives the small ones everything they asked for first, then divides what is left. LIT verified live. Capacity 30 , demands 2 , 2.6 , 4 , 10 , 40 . An equal split gives everyone 6 — which hands 9.4 units to claimants who cannot use them while the one wanting 40 gets 6 . Max-min converges in 3 rounds to 2 , 2.6 , 4 , 10 , 11.4 : everyone below their share is fully satisfied, the remainder goes to the one who can still use it, and all 30 is allocated with none wasted. 2 HOW IT WAS WEAVED · AI + HUMAN Max-min fairness is the allocation behind fair queueing and behind every “fair share” scheduler; the water-filling procedure is the standard construction. AVAN (AI) published the wasted column because it is the one that makes the argument. Equal splitting is not merely less efficient — it is unfair and wasteful at the same time, handing 9.4 units to parties who will not use them while a party that would has to go without. 3 ONE DIMENSION Equal split against max-min, on the same demands. 4 TWO DIMENSIONS · INTERACTIVE Change the capacity and watch the water rise. more capacity ▶ less show equal split reset 5 THREE DIMENSIONS + AVAN’S INVERSE The green forward object: water finding its level. AVAN’s addition (the inverse-companion): the forward reading is that max-min is the fair allocation. The inverse is that it is fair only if asking for less is honest . The whole procedure rewards a small declared demand with full satisfaction, so it is a rule that pays you to understate what you want — and it has no way to tell a claimant who needs 2 from one who asked for 2 to be served first. Read backwards, every fair-share scheduler is an allocation built on self-reported need, and its fairness is exactly as good as that reporting. pause spin LIT a capacity of 30 against demands of 2, 2.6, 4, 10 and 40 gives an equal split of 6 each, which hands 9.4 units to claimants who cannot use them, while max-min converges in 3 rounds to 2, 2.6, 4, 10, 11.4 - everyone below their share fully satisfied, the remainder to the one who can still use it, and all 30 allocated with none wasted FIG Max-min fairness is the allocation behind fair queueing and behind every fair-share scheduler; the water-filling procedure is the standard construction. AVAN published the wasted column because it is the one that makes the argument. Equal splitting is not merely less efficient - it is unfair and wasteful at the same time, handing 9.4 units to parties who will not use them while a party that would has to go without. ◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE SYNC · David Lee Wise (ROOT0), with AVAN