INPUT
hey
my
friend
embedding table
152k × 3584 lookup
x = 3 × 3584
ONE LAYER — identical structure, repeated 28×
shape in = shape out = 3 × 3584
ATTENTION — tokens exchange information
Q 3 × 3584
K 3 × 512
V 3 × 512
computed here, from x, by W_Q / W_K / W_V — one row per token
QKᵀ → softmax
× V → W_O
back to 3 × 3584
+ residual, norm
MLP — each token processed alone
3584
18944
3584
expand
SiLU gate
contract
88% of this layer's weights live here
+ residual, norm
repeat 28× — same structure, new weights every layer
the matrix keeps its 3 × 3584 shape the entire way down
AFTER 28 LAYERS
3 × 3584
keep only the LAST row — it is
the only one that has seen
the whole prompt
1 × 3584
LM head 3584 × 152k
logits 1 × 152,000
one score per vocabulary entry
softmax, sample
hi
ONE token. For the next one,
the whole pass runs again.
← Prev
Next →