THE FOLD / GRIND / BACKPROP / THE CHAIN RULE
THE CHAIN RULE
the error, walked backward through the wires
forward →
x=— → w1=— → a=w1·x=— → w2=— → y=w2·a=— → L=(y−t)²=—
x=— → w1=— → a=w1·x=— → w2=— → y=w2·a=— → L=(y−t)²=—
← backward (the chain rule)
∂L/∂y=— ∂L/∂a=— ∂L/∂w2=— ∂L/∂w1=—
∂L/∂y=— ∂L/∂a=— ∂L/∂w2=— ∂L/∂w1=—
target t 1.0
· η 0.10
step 0 · loss —
LIT A genuine 2-layer net and genuine backprop. Forward: a=w1·x, y=w2·a, L=(y−t)². Backward is the literal chain rule: ∂L/∂y=2(y−t), ∂L/∂a=∂L/∂y·w2, ∂L/∂w2=∂L/∂y·a, ∂L/∂w1=∂L/∂a·x. Each number is recomputed every step from the actual weights; SGD (w←w−η·∂L/∂w) drives the loss toward 0. Change the target or η and it re-solves live.
FIG The glowing wires are decoration; the values on them are real. 'The error walked backward' is how backprop actually works, told as a picture.
FIG The glowing wires are decoration; the values on them are real. 'The error walked backward' is how backprop actually works, told as a picture.
◆ sealed .dlw.fold → folded to ROOT_0 · a sphere of BACKPROP · David Lee Wise (ROOT0), with AVAN