Transformer architecture
The Transformer replaces recurrent sequence processing with attention and position-wise feed-forward computation. Self-attention is more parallelizable than recurrent layers, and the architecture's flexibility helped it spread into vision, audio, and multimodal applications.
Overview. The Transformer replaces recurrent sequence processing with attention and position-wise feed-forward computation. Self-attention is more parallelizable than recurrent layers, and the architecture's flexibility helped it spread into vision, audio, and multimodal applications.
Mechanism and context. The original Transformer composes attention with position-wise feed-forward computation and supplies positional information because attention alone does not encode order. Its token interactions can be computed in parallel during training instead of requiring the recurrent step-by-step path used by earlier sequence models.
Practical use. Trace one block by following tensor shapes through attention, the feed-forward sublayer, and the residual stream. In a deployed model, also record whether the system is encoder-only, decoder-only, or encoder-decoder, because the family label does not specify its training objective or generation behavior.
Related reading. Continue with Scaled dot-product attention for the central equation, Positional encoding for sequence order, Attention complexity for a scaling cost, and Attention Is All You Need for the primary architecture report.
Claims and evidence
Claim ledger, 2The Transformer replaces recurrent sequence processing with attention and position-wise feed-forward computation.
Self-attention is more parallelizable than recurrent layers, and the architecture's flexibility helped it spread into vision, audio, and multimodal applications.