Dot product
The dot product combines coordinate-wise products into a scalar and is used as a similarity score in attention. Its magnitude depends on vector norms, which is why scaled dot-product attention divides by the square root of key dimension.
\mathbf{x}\cdot\mathbf{y}=\sum_i x_i y_i
Overview. The dot product combines coordinate-wise products into a scalar and is used as a similarity score in attention.
Context. Its magnitude depends on vector norms, which is why scaled dot-product attention divides by the square root of key dimension.
Claims and evidence
Claim ledger, 2The dot product combines coordinate-wise products into a scalar and is used as a similarity score in attention.
Its magnitude depends on vector norms, which is why scaled dot-product attention divides by the square root of key dimension.