THE FOLD / CO-OP / THE SYNC / THE CONTENT-DEFINED CHUNK
THE CONTENT-DEFINED CHUNK
cut where the content says, not where the ruler does
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
Cut a file into fixed 4 KB blocks and insert one byte at the front: every block after the insertion shifts by one and nothing matches any more. Cut it where the content says to — at positions where a rolling hash hits a pattern — and the boundaries re-synchronise on their own.
LIT verified live. 200,000 bytes, one byte inserted at offset 50,000. Fixed 4 KB blocking: 49 chunks, of which 37 no longer match — 75.5% of the file must be re-sent for a one-byte edit. Content-defined chunking over the same data: 177 chunks averaging 1,130 bytes, of which exactly 1 changed — 0.6%. The boundary after the edit re-synchronises within a single chunk.
LIT verified live. 200,000 bytes, one byte inserted at offset 50,000. Fixed 4 KB blocking: 49 chunks, of which 37 no longer match — 75.5% of the file must be re-sent for a one-byte edit. Content-defined chunking over the same data: 177 chunks averaging 1,130 bytes, of which exactly 1 changed — 0.6%. The boundary after the edit re-synchronises within a single chunk.
2 HOW IT WAS WEAVED · AI + HUMAN
Andrew Tridgell’s rsync (1996) made the rolling checksum famous; content-defined chunking is the same idea turned into a cut rule, and it underlies every modern deduplicating backup system.
AVAN (AI) measured the resynchronisation directly rather than reasoning about it. The number that matters is not the compression — it is 1. Not "a few", not "roughly one": the damage from an insertion is bounded to the chunk containing it, and the very next boundary is decided by content the edit never touched. The mechanism is the same rolling hash Rabin and Karp made famous; the property measured here is what happens to the boundaries afterwards.
AVAN (AI) measured the resynchronisation directly rather than reasoning about it. The number that matters is not the compression — it is 1. Not "a few", not "roughly one": the damage from an insertion is bounded to the chunk containing it, and the very next boundary is decided by content the edit never touched. The mechanism is the same rolling hash Rabin and Karp made famous; the property measured here is what happens to the boundaries afterwards.
3 ONE DIMENSION
The same file, cut both ways, before and after one inserted byte.
4 TWO DIMENSIONS · INTERACTIVE
Move the edit and watch what survives.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: a boundary that finds itself again.
AVAN’s addition (the inverse-companion): the forward reading is that content-defined chunking is robust to insertion. The inverse is that fixed blocking was never storing your file — it was storing your file plus an offset, and the offset was the fragile part. Nothing about the bytes changed; 199,999 of 200,000 are identical. What broke was the coordinate system laid over them. Read backwards, most of what we call a diff is a disagreement about where to start counting, and the fix is never to count from the outside.
LIT 200,000 bytes with one byte inserted at offset 50,000: fixed 4 KB blocking gives 49 chunks of which 37 no longer match, 75.5% of the file re-sent for a one-byte edit, while content-defined chunking over the same data gives 177 chunks averaging 1,130 bytes of which exactly 1 changed - 0.6%, the boundary re-synchronising within a single chunk
FIG Andrew Tridgell's rsync (1996) made the rolling checksum famous; content-defined chunking is the same idea turned into a cut rule and it underlies every modern deduplicating backup system. AVAN measured the resynchronisation directly rather than reasoning about it. The number that matters is not the compression but the 1: the damage from an insertion is bounded to the chunk containing it, and the next boundary is decided by content the edit never touched.
FIG Andrew Tridgell's rsync (1996) made the rolling checksum famous; content-defined chunking is the same idea turned into a cut rule and it underlies every modern deduplicating backup system. AVAN measured the resynchronisation directly rather than reasoning about it. The number that matters is not the compression but the 1: the damage from an insertion is bounded to the chunk containing it, and the next boundary is decided by content the edit never touched.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE SYNC · David Lee Wise (ROOT0), with AVAN