THE FOLD / SPAWN / COLD BOOT / THE BYTE ORDER MARK

THE BYTE ORDER MARK

metadata living inside the data

1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION

A mark at the front of a file saying which end of a number comes first. UTF-8 has no ends to order. The mark got used anyway, as a label, and it is invisible.

LIT verified live. the byte order mark is U+FEFF, encoded in UTF-8 as the three bytes EF BB BF. It belongs to Unicode’s format category, so it draws nothing. Put it in front of a JSON document and JSON.parse throws; strip it and the identical document parses. The string "abc" with a leading mark has length 4 rather than 3 and is not equal to "abc" — and a whitespace trim happens to remove it, which means the bug appears and disappears depending on whether some earlier stage trimmed.

2 HOW IT WAS WEAVED · AI + HUMAN

The BOM is required for UTF-16, optional and discouraged for UTF-8, and emitted by default by several Windows editors — which is why it is usually met as an unexplained parse error in a file that looks fine.

AVAN (AI) made the JSON failure the measurement rather than the anecdote: the same bytes, minus three at the front, parse. That is the whole diagnosis and it is one line. The trim result is the part worth carrying — a bug that is removed by an unrelated cleanup step is a bug that will be reported as intermittent, and intermittent is what a defect looks like when the pipeline has more stages than the report mentions.

3 ONE DIMENSION

Three bytes at the front. Nothing on the screen.

4 TWO DIMENSIONS · INTERACTIVE

Put the mark in front of a document and parse it.

5 THREE DIMENSIONS + AVAN’S INVERSE

The green forward object.

AVAN’s addition (the inverse-companion): the forward reading is to strip the BOM on input. The inverse is that it is a label pretending to be content. Every other piece of metadata about a file — its name, its type, its length — lives outside the bytes; this one was put inside them, so every reader must know to skip it and none of them can be told by the file itself. Read backwards, the mark is not the problem; putting metadata in the same channel as data is the problem, and this is simply the smallest possible example of it.

LIT the byte order mark is U+FEFF, encoded in UTF-8 as the three bytes EF BB BF, and it belongs to Unicode's format category so it draws nothing; put it in front of a JSON document and JSON.parse throws while stripping it makes the identical document parse, and the string abc with a leading mark has length 4 rather than 3 and is not equal to abc - and a whitespace trim happens to remove it, so the bug appears and disappears depending on whether some earlier stage trimmed

FIG The BOM is required for UTF-16, optional and discouraged for UTF-8, and emitted by default by several Windows editors - which is why it is usually met as an unexplained parse error in a file that looks fine. AVAN made the JSON failure the measurement rather than the anecdote: the same bytes, minus three at the front, parse. The trim result is the part worth carrying - a bug removed by an unrelated cleanup step gets reported as intermittent, and intermittent is what a defect looks like when the pipeline has more stages than the report mentions.

◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of COLD BOOT · David Lee Wise (ROOT0), with AVAN