Transformer architecture
The Transformer replaces recurrent sequence processing with attention and position-wise feed-forward computation. Parallel training and flexible token interactions helped the architecture become a foundation for language and multimodal systems.
Overview. The Transformer replaces recurrent sequence processing with attention and position-wise feed-forward computation. Parallel training and flexible token interactions helped the architecture become a foundation for language and multimodal systems.
Mechanism and context. The original Transformer composes attention with position-wise feed-forward computation and supplies positional information because attention alone does not encode order. Its token interactions can be computed in parallel during training instead of requiring the recurrent step-by-step path used by earlier sequence models.
Practical use. Trace one block by following tensor shapes through attention, the feed-forward sublayer, and the residual stream. In a deployed model, also record whether the system is encoder-only, decoder-only, or encoder-decoder, because the family label does not specify its training objective or generation behavior.
Related reading. Continue with Scaled dot-product attention for the central equation, Positional encoding for sequence order, Attention complexity for a scaling cost, and Attention Is All You Need for the primary architecture report.
Claims and evidence
Claim ledger, 2The Transformer replaces recurrent sequence processing with attention and position-wise feed-forward computation.
Parallel training and flexible token interactions helped the architecture become a foundation for language and multimodal systems.