THE FOLD / CHEAT / THE EXPLOIT / THE UTF-8 OVERLONG

THE UTF-8 OVERLONG

384 spare spellings of 128 characters

1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION

There is exactly one correct way to encode a character in UTF-8, and several that also work. A decoder that accepts the extras will hand you a slash you did not see coming.

LIT verified live. each of the 128 ASCII code points has 3 non-minimal encodings — padded out to two, three and four bytes — giving 384 overlong forms. A decoder that simply reassembles the bits accepts all 384 and returns the original character every time. The browser’s standards-conforming decoder accepts 0 of them and returns the replacement character instead. The slash /, U+002F, has the overlong forms C0 AF, E0 80 AF and F0 80 80 AF — none of which contains the byte 0x2F that a path filter is looking for.

2 HOW IT WAS WEAVED · AI + HUMAN

Overlong UTF-8 was the mechanism behind the IIS directory-traversal worms of 2001; the Unicode standard made non-minimal forms illegal precisely because filters and decoders disagreed about them.

AVAN (AI) ran both decoders — a hand-written naive one and the browser’s TextDecoder — rather than describing the difference, so 384 against 0 is measured on the same inputs. The point is not that the naive decoder is badly written. It is that it is the obvious one: reassemble the bits and you get the right character, which is exactly the behaviour that makes the check upstream meaningless.

3 ONE DIMENSION

384 spare spellings of 128 characters.

4 TWO DIMENSIONS · INTERACTIVE

Encode a character the wrong number of ways.

5 THREE DIMENSIONS + AVAN’S INVERSE

The green forward object.

AVAN’s addition (the inverse-companion): the forward reading is that decoders must reject non-minimal forms. The inverse is that the bug is never in the decoder, it is in the gap between two of them. A filter reads bytes and looks for 0x2F; a decoder reads bytes and produces characters; and the attack lives in the fact that these two answered the same question differently and neither was wrong on its own. Read backwards, every validate-then-transform pipeline has this shape, and the defence is not a better filter but refusing to validate anything before it is in its final form.

LIT each of the 128 ASCII code points has 3 non-minimal encodings padded out to two, three and four bytes, giving 384 overlong forms, and a decoder that simply reassembles the bits accepts all 384 and returns the original character every time while the browser's standards-conforming decoder accepts 0 of them; the slash U+002F has the overlong forms C0 AF, E0 80 AF and F0 80 80 AF, none of which contains the byte 0x2F that a path filter is looking for

FIG Overlong UTF-8 was the mechanism behind the IIS directory-traversal worms of 2001; the Unicode standard made non-minimal forms illegal precisely because filters and decoders disagreed about them. AVAN ran both decoders - a hand-written naive one and the browser's TextDecoder - so 384 against 0 is measured on the same inputs. The point is not that the naive decoder is badly written; it is that it is the obvious one, and that is what makes the check upstream meaningless.

◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of THE EXPLOIT · David Lee Wise (ROOT0), with AVAN