Attention complexity
Dense self-attention constructs interactions for every pair of positions in a sequence. Its score matrix therefore grows quadratically with sequence length before implementation-specific optimizations.
\mathcal{O}(T^2d)
Overview. Dense self-attention constructs interactions for every pair of positions in a sequence.
Context. Its score matrix therefore grows quadratically with sequence length before implementation-specific optimizations.
Claims and evidence
Claim ledger, 2Dense self-attention constructs interactions for every pair of positions in a sequence.
Its score matrix therefore grows quadratically with sequence length before implementation-specific optimizations.