Transformer architecture history

Overview. The Transformer replaces recurrent sequence processing with attention and position-wise feed-forward computation. Self-attention is more parallelizable than recurrent layers, and the architecture's flexibility helped it spread into vision, audio, and multimodal applications.

Mechanism and context. The original Transformer composes attention with position-wise feed-forward computation and supplies positional information because attention alone does not encode order. Its token interactions can be computed in parallel during training instead of requiring the recurrent step-by-step path used by earlier sequence models.

Practical use. Trace one block by following tensor shapes through attention, the feed-forward sublayer, and the residual stream. In a deployed model, also record whether the system is encoder-only, decoder-only, or encoder-decoder, because the family label does not specify its training objective or generation behavior.

Related reading. Continue with Scaled dot-product attention for the central equation, Positional encoding for sequence order, Attention complexity for a scaling cost, and Attention Is All You Need for the primary architecture report.

Overview. The Transformer replaces recurrent sequence processing with attention and position-wise feed-forward computation. Parallel training and flexible token interactions helped the architecture become a foundation for language and multimodal systems.

Mechanism and context. The original Transformer composes attention with position-wise feed-forward computation and supplies positional information because attention alone does not encode order. Its token interactions can be computed in parallel during training instead of requiring the recurrent step-by-step path used by earlier sequence models.

Practical use. Trace one block by following tensor shapes through attention, the feed-forward sublayer, and the residual stream. In a deployed model, also record whether the system is encoder-only, decoder-only, or encoder-decoder, because the family label does not specify its training objective or generation behavior.

Related reading. Continue with Scaled dot-product attention for the central equation, Positional encoding for sequence order, Attention complexity for a scaling cost, and Attention Is All You Need for the primary architecture report.

Overview. The Transformer replaces recurrent sequence processing with attention and position-wise feed-forward computation. Parallel training and flexible token interactions helped the architecture become a foundation for language and multimodal systems.

Mechanism and context. The original Transformer composes attention with position-wise feed-forward computation and supplies positional information because attention alone does not encode order. Its token interactions can be computed in parallel during training instead of requiring the recurrent step-by-step path used by earlier sequence models.

Practical use. Trace one block by following tensor shapes through attention, the feed-forward sublayer, and the residual stream. In a deployed model, also record whether the system is encoder-only, decoder-only, or encoder-decoder, because the family label does not specify its training objective or generation behavior.

Limitations and evidence. The 2017 paper establishes an architecture and reports results for its stated tasks and setup; it does not make every later Transformer equivalent. Context length, attention variant, training data, objective, and implementation all remain part of a current model claim.

Related reading. Continue with Scaled dot-product attention for the central equation, Positional encoding for sequence order, Attention complexity for a scaling cost, and Attention Is All You Need for the primary architecture report.

The Transformer replaces recurrent sequence processing with attention and position-wise feed-forward computation.

Parallel training and flexible token interactions helped the architecture become a foundation for language and multimodal systems.

Why it matters: Transformer architecture is best understood as part of a system, not as a label in isolation. The practical questions are what inputs it accepts, what outputs it produces, which evidence supports its capabilities, what constraints surround deployment, and how those details change over time. The Compendium separates the documented facts above from interpretation and records an as-of date so readers can re-check the primary source.

Reading note: this is original explanatory writing. It links to the cited source instead of reproducing protected text. Product availability, model names, pricing, and policies can change; follow the source for the provider's current position.

The Transformer replaces recurrent sequence processing with attention and position-wise feed-forward computation.

Parallel training and flexible token interactions helped the architecture become a foundation for language and multimodal systems.

Why it matters: Transformer architecture is best understood as part of a system, not as a label in isolation. The practical questions are what inputs it accepts, what outputs it produces, which evidence supports its capabilities, what constraints surround deployment, and how those details change over time. The Atlas separates the documented facts above from interpretation and records an as-of date so readers can re-check the primary source.

Reading note: this is original explanatory writing. It links to the cited source instead of reproducing protected text. Product availability, model names, pricing, and policies can change; follow the source for the provider's current position.