Embedder, opened up

counts → PPMI → Gram → eigenvectors → 8 components LIT · LIVE

step 0observations 0 coverage 2/1080⟨AB⟩ — re-embed every 25 steps

A · COUNTS M

What it is. Raw tally. M[i][j] = how many times letter j followed letter i on the agents' path. Nothing normalised. Frequent letters dominate every row, so a bright cell mostly means "both letters are common" — not that they belong together.

B · PPMI S

What it does. log( P(i,j) / P(i)P(j) ), negatives clipped to zero. Divides out how common each letter is on its own, leaving only surprise. A pair scores high when it co-occurs more than independence predicts. This is what stops the matrix from just being a frequency ranking.

C · GRAM S·Sᵀ

What it does. G[i][j] = dot product of row i and row j. Two letters score high when they are followed by the same things — not when they follow each other. This is the step that makes similarity distributional. Symmetrise the wrong matrix here and the whole thing inverts.

D · SPECTRUM

What it does. Eigenvalues of G, largest first. Each one is how much variance its component explains. The drop-off tells you how many dimensions carry real structure — everything past the elbow is mostly sampling noise at this corpus size.

What an embedding actually is, in one line: a change of coordinates. The counts describe each letter by identity — 27 numbers saying which specific letters follow it. The embedding describes each letter by position — 8 numbers saying where it sits in a space built from the whole table at once. Identity can only compare things that literally co-occurred. Position can compare things that never met.

Each component window shows one axis. The 27 letters are placed along that single eigenvector, and the axis is scored against four probes: is it separating vowels from consonants, common from rare, word-initial from not, word-final from not. Correlation |r| ≥ 0.45 gets a green tag. Anything weaker gets flagged mixed.

The expected result on the full corpus: component 2 tracks vowels at r = 0.64, component 8 tracks word-initial at 0.52, component 5 tracks word-final at 0.47. The other five have no clean interpretation at all.

That is not a defect, it is the point. Five of eight axes encode combinations of features rather than any single one. There is no vowel dimension — there is a vowel direction, spread across several axes, and each axis carries fragments of several unrelated properties. Superposition, small enough to see the whole of it.