Scaled dot-product attention
Scaled dot-product attention forms weights from query-key products and uses them to combine value vectors. The scale factor counters growth in dot-product variance as key dimension increases.
Overview. Scaled dot-product attention forms weights from query-key products and uses them to combine value vectors. The scale factor counters growth in dot-product variance as key dimension increases.
Mechanism and context. Queries are compared with keys by matrix multiplication, the scores are scaled by the square root of key dimension, softmax turns them into weights, and those weights combine values. Keeping Q, K, and V distinct explains how a position can decide what to read separately from the information it receives.
Practical use. Annotate the equation with batch, head, sequence, and feature dimensions before implementing it. That makes shape errors visible and shows where masks apply, where the sequence-by-sequence score matrix is created, and which axis softmax must normalize.
Related reading. Read Dot product and Softmax for the component operations, Self-attention for the sequence-level concept, Attention complexity for its dense cost, and Transformer architecture for the surrounding block.
Claims and evidence
Claim ledger, 2Scaled dot-product attention forms weights from query-key products and uses them to combine value vectors.
The scale factor counters growth in dot-product variance as key dimension increases.