Before the introduction of the transformer, sequence transduction tasks such as machine translation and language modeling relied primarily on recurrent neural networks (RNNs), particularly long short-term memory (LSTM) networks and gated recurrent units (GRUs).¹⁶¹⁷¹⁸ These architectures processed sequences token by token, updating an internal hidden state at each time step based on the previous state and the current input.¹⁹ While effective for short sequences, recurrent models suffered from two structural limitations: the sequential dependency across time steps prevented parallel computation within training examples, and information from early tokens tended to degrade or vanish over long sequences as it was compressed into fixed-length state vectors.

To alleviate the fixed-vector bottleneck in encoder-decoder RNNs, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio introduced an additive attention mechanism for neural machine translation in 2014, followed by multiplicative attention variants proposed by Minh-Thang Luong and colleagues in 2015.²⁰²¹²² These mechanisms allowed a recurrent decoder to dynamically reference and weight all hidden states of the encoder at each output step rather than relying solely on the final encoder state. However, because these attention layers were still paired with underlying recurrent or convolutional networks—such as in ByteNet and ConvS2S—training remained constrained by sequential operations or multi-layer convolutional paths.²³

In June 2017, a team of eight researchers from Google Brain, Google Research, and the University of Toronto—Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin—released the paper "Attention Is All You Need" on arXiv and presented it at the Conference on Neural Information Processing Systems (NIPS, later NeurIPS) in Long Beach, California.²⁴ The authors proposed replacing recurrence and convolutions entirely with stacked self-attention and point-wise feed-forward layers. On the WMT 2014 English-to-German translation benchmark, the resulting model achieved a BLEU score of 28.4, surpassing the best existing ensembles by more than 2.0 BLEU points. On the WMT 2014 English-to-French task, it reached a single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, representing a fraction of the compute cost required by earlier recurrent architectures.

Following the 2017 publication, the machine learning community rapidly adopted the transformer for transfer learning. Instead of training models from scratch on task-specific datasets, researchers began pre-training large transformers on vast corpora of unlabeled text using self-supervised objectives, followed by fine-tuning on downstream tasks.²⁵ In 2018, OpenAI introduced the first Generative Pre-trained Transformer (GPT) using a decoder-only setup, while Google introduced BERT (Bidirectional Encoder Representations from Transformers) using an encoder-only setup. As hardware capacity expanded, scaling laws demonstrated that transformer performance improved predictably with increases in parameter count, dataset size, and training compute, culminating in general-purpose large language models.

A transformer processes data by converting discrete inputs into continuous vector representations and passing them through a series of stacked layers.²⁶ In the standard baseline model described in 2017, the network consists of six identical encoder layers and six identical decoder layers, operating on internal vector representations with a model dimension of 512.²⁷²⁸

As the transformer was adapted to diverse machine learning problems, practitioners separated the original encoder-decoder structure into three primary architectural paradigms: encoder-only, decoder-only, and encoder-decoder models.

Encoder-only transformers discard the decoder stack and utilize only bidirectional self-attention layers, allowing every token to attend to both preceding and succeeding tokens across the entire input sequence. Often referred to as auto-encoding models, they are typically pre-trained via masked language modeling, in which random tokens in the input are masked and the network learns to reconstruct them from surrounding context. Prominent examples include Google's BERT, RoBERTa, and DistilBERT. Because they construct comprehensive representations of full sequences, encoder-only models are widely deployed for text classification, sentiment analysis, named entity recognition, extractive question answering, and generating dense vector embeddings for information retrieval and search engines.

Decoder-only transformers omit the encoder and cross-attention layers, relying entirely on causally masked self-attention blocks. Pre-trained autoregressively to predict the next token in a sequence given all preceding tokens, decoder-only architectures treat both the user prompt and the generated response as a single continuous sequence. Because any natural language task can be framed as text continuation without task-specific architectural heads, decoder-only models became the standard design for large language models, including OpenAI's GPT series, Meta's Llama family, and Anthropic's Claude.

Encoder-decoder transformers retain both stacks of the original 2017 design along with cross-attention. They are tailored for sequence-to-sequence tasks where an input sequence is mapped to a structurally distinct output sequence, such as machine translation, document summarization, and speech-to-text transcription. Notable encoder-decoder models include Google's T5 (Text-to-Text Transfer Transformer), which frames all text processing tasks as sequence-to-sequence conversions, and Facebook's BART (Bidirectional and Auto-Regressive Transformers).

Between 2017 and the mid-2020s, empirical experimentation on large-scale language models led researchers to replace several design choices of the original Vaswani et al. paper, converging on a standardized modern transformer recipe optimized for training stability and inference speed.³⁷

A primary shift involved the placement and mathematical formulation of layer normalization. The 2017 transformer used a post-normalization ("Post-LN") layout, placing layer normalization directly on the residual path after the addition of the sublayer output, which often required a careful learning-rate warmup phase to prevent gradient instability in deep networks. Modern transformers almost universally adopt pre-normalization ("Pre-LN"), applying normalization inside the residual branch before the attention and feed-forward operations so that gradients flow unimpeded through the main residual stream. Furthermore, standard LayerNorm—which computes both mean and variance—has largely been superseded by Root Mean Square Layer Normalization (RMSNorm), which drops the mean-centering operation and bias terms to reduce computational overhead without sacrificing model accuracy.³⁸

Positional representations and feed-forward activations underwent similar standardization. Rather than adding absolute sinusoidal encodings to the initial token embeddings, most contemporary models use Rotary Position Embedding (RoPE), which encodes relative token distances by rotating the query and key vectors in complex vector space at every attention layer, improving length extrapolation.³⁹ In the feed-forward sublayers, the original ReLU activation was replaced first by the Gaussian Error Linear Unit (GELU) and subsequently by Gated Linear Unit variants such as SwiGLU, which pairs a Swish activation with a learnable gating linear projection.

To accelerate autoregressive inference, attention mechanisms were modified to reduce the memory footprint of the key-value (KV) cache—the stored key and value tensors of past tokens that prevent recomputation at each generation step.⁴⁰ Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) share single key and value heads across multiple query heads, shrinking KV-cache memory bandwidth requirements by factors of four to eight while preserving quality close to standard multi-head attention. Additionally, frontier models frequently replace dense feed-forward networks with sparse Mixture-of-Experts (MoE) layers, where a learned router activates only a subset of specialized feed-forward sub-networks ("experts") for each token, decoupling total parameter count from per-token computational cost.

Although conceived for machine translation, the transformer's capacity to model pairwise relationships across arbitrary token sequences enabled its expansion into virtually all domains of artificial intelligence. In natural language processing, transformers displaced recurrent networks across conversational chatbots, automated code generation, abstractive summarization, and semantic search.

In computer vision, where convolutional neural networks (CNNs) such as ResNet had long been dominant, Alexey Dosovitskiy and colleagues at Google introduced the Vision Transformer (ViT) in 2020.⁴¹⁴² A Vision Transformer divides an image into a grid of fixed-size square patches, flattens each patch into a vector via a linear projection, adds positional embeddings, and feeds the resulting sequence into a standard transformer encoder. Variants such as the Swin Transformer introduced hierarchical representations and shifted window attention, achieving state-of-the-art performance in image classification, object detection, and semantic segmentation.⁴³ Transformers also serve as the backbone for diffusion-based image and video generation models, replacing earlier convolutional U-Net architectures.

In the natural sciences and sequential decision-making, transformers have been applied to biological sequences, audio signals, and robotics. DeepMind's AlphaFold 2, introduced by John Jumper and colleagues in 2020 during the 14th Critical Assessment of Protein Structure Prediction (CASP14), utilized a specialized transformer module called the Evoformer to apply row-wise and column-wise self-attention over multiple sequence alignments and pairwise amino acid representations, predicting 3D protein structures to within atomic accuracy. In audio and multimodal processing, transformer architectures process acoustic frames alongside text and visual tokens inside unified models, while in reinforcement learning and robotics, systems such as the Decision Transformer cast control trajectories of states, actions, and rewards as sequence modeling problems.

The principal technical limitation of the standard transformer architecture is the quadratic time and memory complexity—$\mathcal{O}(n^2)$—of its self-attention mechanism with respect to sequence length $n$.⁴⁴ Because every token computes an attention score against every other token in the context window, doubling the length of an input document quadruples the required attention compute and memory footprint. During long-context inference or multi-turn interactions, storing the expanding KV cache in high-bandwidth GPU memory creates severe latency and economic bottlenecks. Furthermore, training foundation-scale transformers requires massive distributed GPU clusters, drawing scrutiny over financial costs and energy consumption.

To mitigate hardware bottlenecks without altering the mathematical output of standard attention, Tri Dao and colleagues introduced FlashAttention, an I/O-aware exact attention algorithm that uses tiling and online softmax recomputation to keep intermediate attention matrices inside fast on-chip GPU static random-access memory (SRAM) rather than writing them to slower high-bandwidth memory (HBM). Other transformer-preserving optimizations compress or quantize the KV cache, such as Multi-head Latent Attention (MLA) introduced in DeepSeek-V2, or restrict full attention to local sliding windows combined with periodic global attention layers.

A parallel line of research seeks to replace quadratic self-attention entirely with sub-quadratic, linear-time sequence models. Prominent among these are structured state space models (SSMs), such as Mamba—introduced in December 2023 by Albert Gu and Tri Dao at Carnegie Mellon University and Princeton University—and its successors Mamba-2 and Mamba-3. By compressing prior context into a fixed-size recurrent state controlled by input-dependent selection mechanisms, Mamba achieves linear scaling with sequence length and constant-memory inference without a growing KV cache. Because pure state space models can struggle with exact associative recall over long contexts compared to full attention, many production architectures have converged on hybrid designs—such as Jamba and RecurrentGemma—that interleave standard transformer self-attention layers with linear-time state space or recurrent layers.