Gaussian Error Linear Unit
GELU gates an input smoothly according to the Gaussian cumulative distribution. It differs from ReLU's hard zero threshold and is widely used in Transformer feed-forward blocks.
\operatorname{GELU}(x)=x\Phi(x)
Overview. GELU gates an input smoothly according to the Gaussian cumulative distribution.
Context. It differs from ReLU's hard zero threshold and is widely used in Transformer feed-forward blocks.
Claims and evidence
Claim ledger, 2GELU gates an input smoothly according to the Gaussian cumulative distribution.
It differs from ReLU's hard zero threshold and is widely used in Transformer feed-forward blocks.