THE FOLD / RESPAWN / GARBAGE COLLECTION / THE CROSS ENTROPY
THE CROSS ENTROPY
the bits you pay for being wrong
1 WHAT IT IS · WHAT IT DOES · FACT OR FICTION
Entropy is the shortest average message length achievable if you know the true distribution. Cross-entropy is what you actually pay when you encode using the wrong distribution, and the difference between them is the KL divergence — the surcharge, in bits, for believing something false. It is never negative, and it is zero only if your model is exactly right. This is the loss function almost every neural network is trained on, and the number it reports is a literal price in bits.
LIT verified live: for a 12-symbol source, H(p) = 2.738243 bits, H(p,q) = 3.312595 and KL(p‖q) = 0.574352, with H(p,q) − H(p) − KL equal to 0 exactly. Encoding 120,000 actual draws costs 3.314950 bits per symbol with the wrong model and 2.749046 with the right one — a measured surcharge of 0.5659 bits against a predicted 0.5744. Across 2,000 random models there are 0 negative divergences, KL(p‖p) is 0 exactly, and the divergence is not symmetric: 0.5744 one way, 0.6171 the other.
LIT verified live: for a 12-symbol source, H(p) = 2.738243 bits, H(p,q) = 3.312595 and KL(p‖q) = 0.574352, with H(p,q) − H(p) − KL equal to 0 exactly. Encoding 120,000 actual draws costs 3.314950 bits per symbol with the wrong model and 2.749046 with the right one — a measured surcharge of 0.5659 bits against a predicted 0.5744. Across 2,000 random models there are 0 negative divergences, KL(p‖p) is 0 exactly, and the divergence is not symmetric: 0.5744 one way, 0.6171 the other.
2 HOW IT WAS WEAVED · AI + HUMAN
David (human) seated this at GARBAGE COLLECTION: the KL gap is exactly the wasted bits, and nothing else in the calculation is waste.
AVAN (AI) measured the code length by actually spending it rather than by evaluating the formula twice. Drawing 120,000 symbols from p and summing −log₂q(x) is the ideal code length an arithmetic coder would pay, and its average converges to H(p,q) by definition — so agreement to three decimals is a check on the identity, not a restatement of it. Huffman coding was deliberately not used for this: it is only guaranteed within 1 bit of the entropy, so it would have introduced a discrepancy that has nothing to do with the claim and would need explaining away. The asymmetry is worth dwelling on, because it is why “distance” is the wrong word: KL(p‖q) punishes assigning low probability to things that happen, and KL(q‖p) punishes something else entirely.
AVAN (AI) measured the code length by actually spending it rather than by evaluating the formula twice. Drawing 120,000 symbols from p and summing −log₂q(x) is the ideal code length an arithmetic coder would pay, and its average converges to H(p,q) by definition — so agreement to three decimals is a check on the identity, not a restatement of it. Huffman coding was deliberately not used for this: it is only guaranteed within 1 bit of the entropy, so it would have introduced a discrepancy that has nothing to do with the claim and would need explaining away. The asymmetry is worth dwelling on, because it is why “distance” is the wrong word: KL(p‖q) punishes assigning low probability to things that happen, and KL(q‖p) punishes something else entirely.
3 ONE DIMENSION
Entropy, cross-entropy, and the gap you pay in bits.
4 TWO DIMENSIONS · INTERACTIVE
Move the model away from the truth and watch the surcharge appear.
5 THREE DIMENSIONS + AVAN’S INVERSE
The green forward object: the simplex of models, with the surcharge as height above the truth.
AVAN’s addition (the inverse-companion): the forward reading is “cross-entropy measures how wrong your model is.” The inverse is that it measures how wrong your model is about the things that actually happen, and is entirely indifferent to the rest. Every term is weighted by p, so a model can be arbitrarily deranged about events of probability zero and pay nothing at all. That is why the asymmetry exists and why it matters which way round you train: minimising KL(p‖q) makes q cover everything p does, and minimising KL(q‖p) lets q pick one mode and ignore the others. Read backwards, the loss function is not measuring truth — it is measuring usefulness under a fixed sampling of the world, and it will never charge you for a question nobody asks.
LIT for a 12-symbol source, H(p) = 2.738243 bits, H(p,q) = 3.312595 and KL(p||q) = 0.574352, with H(p,q) - H(p) - KL equal to 0 exactly; encoding 120,000 actual draws costs 3.314950 bits per symbol with the wrong model and 2.749046 with the right one, a measured surcharge of 0.5659 against a predicted 0.5744; across 2,000 random models there are 0 negative divergences, KL(p||p) is 0 exactly, and the divergence is not symmetric - 0.5744 one way, 0.6171 the other
FIG The code length was measured by ACTUALLY SPENDING IT rather than by evaluating the formula twice. Drawing 120,000 symbols from p and summing -log2 q(x) is the ideal code length an arithmetic coder would pay, and its average converges to H(p,q) by definition, so agreement to three decimals is a check on the identity rather than a restatement of it. Huffman coding was deliberately NOT used: it is only guaranteed within 1 bit of the entropy, so it would introduce a discrepancy with nothing to do with the claim. The asymmetry is why 'distance' is the wrong word.
FIG The code length was measured by ACTUALLY SPENDING IT rather than by evaluating the formula twice. Drawing 120,000 symbols from p and summing -log2 q(x) is the ideal code length an arithmetic coder would pay, and its average converges to H(p,q) by definition, so agreement to three decimals is a check on the identity rather than a restatement of it. Huffman coding was deliberately NOT used: it is only guaranteed within 1 bit of the entropy, so it would introduce a discrepancy with nothing to do with the claim. The asymmetry is why 'distance' is the wrong word.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of GARBAGE COLLECTION · David Lee Wise (ROOT0), with AVAN