domain 03 / 64 · RULES OF ATTRACTION · the attention heads

THE RULES OF ATTRACTION

attention, read as attraction: in a transformer every token pulls the model's gaze by a different RULE, and each rule is a documented type of ATTENTION HEAD.

21 heads · each a query×key signature · sealed & live

HONEST: these render DOCUMENTED interpretability findings (induction/IOI/successor/sink and kin, from the mech-interp literature) as interactive toys — most are amber tier (modeled structure, not a fresh measurement of a live model), each two-layer honest about what is real vs figure. The title

▦ ATTENTION HEAD MAP · a query cursor sweeps every head

HEADS · —

hover a head · click to enter its sphere

▦ the heads

every head, named & enterable

THE SOFTMAXattraction · turn similarity scores into weights that sum to→THE POSITIONAL ENCODINGattraction · attention has no order — stamp each slot with a→THE QK CIRCUITattraction · query · key / √d — each token attracts its most→THE CAUSAL MASKattraction · no peeking ahead — the future is set to zero be→THE MULTI-HEAD ATTENTIONattraction · run many attention heads in parallel subspaces,→THE VALUE MIXINGattraction · the output is a convex blend of the values it a→THE ATTENTION ENTROPYattraction · one number for how focused a head is — 0 (point→INDUCTION HEAD · INDthe echo · match-then-copy · the engine of in-context learni→THE SUCCESSOR HEADattraction · ordinal → next ordinal (Gould et al. 2023)→THE DUPLICATE-TOKEN HEADattraction · attend back to the same token — the first move →THE S-INHIBITION HEADattraction · suppress the repeated subject so the right name→THE SKIP-TRIGRAMattraction · [A]…[B]→[C], all a one-layer attention model ca→ATTENTION SINK · SNKthe pressure valve · dump attention on the first token→PREVIOUS-TOKEN HEAD · PRVattend to i−1 · stage one of the induction circuit→RETRIEVAL HEAD · RETreach far back · copy the span the answer needs→COPY SUPPRESSION · CSPraise — or LOWER — an attended token's logit→NAME-MOVER HEAD · NMVthe IOI circuit · move the owed name to the end→SUCCESSOR HEAD · SUCincrement an ordered set · Monday→Tuesday, 4→5→FAITHFULNESS HEAD · FTHdoes the stated reasoning match the real computation→COHERENCE HEAD · COHhold one language on multilingual input→POSITIONAL / SYNTACTIC · POSattend by place or grammar, not by content→