Before the emergence of neural language models, natural language processing relied on symbolic rules and statistical n-gram models, which estimated word probabilities based on short preceding sequences. In the 2010s, researchers adopted recurrent neural networks (RNNs) and long short-term memory (LSTM) networks, alongside distributed word representations that mapped vocabulary into continuous vector spaces.¹⁰ Although systems such as ELMo (Embeddings from Language Models) demonstrated the value of contextualized representations, sequential architectures processed text word by word, creating computational bottlenecks and struggling to retain context across long passages.

A foundational shift occurred in 2017 when researchers at Google Brain published the paper "Attention Is All You Need," introducing the transformer architecture. By replacing recurrence with self-attention mechanisms, transformers could evaluate relationships between all words in a sequence simultaneously, allowing training to scale efficiently across graphics processing units (GPUs).¹¹ In 2018, two influential transformer-based models established the paradigm of generative pre-training followed by task-specific fine-tuning. Google released BERT (Bidirectional Encoder Representations from Transformers), a 340-million-parameter model that analyzed context in both directions and became a standard for search and classification tasks. That same year, OpenAI introduced GPT-1, demonstrating that a decoder-only transformer trained to predict the next word on diverse, unlabeled text could transfer its learning to varied downstream benchmarks.

Between 2019 and 2022, researchers focused on expanding parameter counts and training datasets. OpenAI's GPT-2 in 2019 (1.5 billion parameters) and GPT-3 in 2020 (175 billion parameters) showed that scale alone unlocked "in-context learning," enabling a model to solve new tasks from a few examples provided in the prompt without updating its internal weights. Other organizations trained competing foundation models, including Google's 540-billion-parameter PaLM and DeepMind's Gopher and Chinchilla. The release of OpenAI's ChatGPT in November 2022, which paired a GPT-3.5 model with alignment techniques to follow conversational instructions, brought large language models to mass public adoption and triggered intense commercial competition.¹

From 2023 onward, the field expanded beyond pure text generation toward native multimodality, open-weight ecosystems, and deliberate reasoning.¹²¹³ Models such as GPT-4 and GPT-4o integrated text, audio, and visual processing, while Meta's Llama series, Mistral, and Alibaba's Qwen established a robust ecosystem of openly available model weights. In late 2024 and early 2025, the launch of OpenAI's o1 and o3 series alongside Chinese laboratory DeepSeek's open-source DeepSeek-R1 shifted industry focus toward models trained via reinforcement learning to perform extended, step-by-step reasoning before producing an answer. By 2025 and 2026, hybrid systems such as Anthropic's Claude 3.7 Sonnet and subsequent frontier releases combined rapid conversational responses with controllable test-time compute and autonomous tool execution.¹⁴

Large language models do not process raw text directly as characters or whole words; instead, incoming text first passes through a preprocessor called a tokenizer.¹⁵¹⁶ Most contemporary models rely on subword tokenization algorithms such as Byte-Pair Encoding (BPE), WordPiece, or SentencePiece.¹⁷ In Byte-Pair Encoding, the algorithm begins with individual characters or bytes and iteratively merges the most frequently co-occurring pairs in a training corpus until a fixed vocabulary size is reached. Consequently, common words remain intact as single tokens, while rarer or complex words are split into meaningful roots, prefixes, and suffixes.¹⁸ In English, a token corresponds on average to roughly four characters or three-quarters of a word, and each token is assigned a unique integer identifier that maps to a high-dimensional numerical vector known as an embedding.

Once converted into token embeddings and combined with positional encodings—such as Rotary Position Embeddings (RoPE) or ALiBi—that indicate word order, the sequence enters a stack of transformer layers. Each layer contains two primary components: a multi-head self-attention mechanism and a feed-forward neural network. The self-attention mechanism calculates mathematical compatibility scores between every pair of tokens within the model's context window, allowing the representation of an ambiguous word, such as "bank," to dynamically incorporate information from surrounding words that clarify whether it refers to a financial institution or a river.¹⁹ Hardware-aware algorithms such as FlashAttention and Grouped-Query Attention have been widely adopted to reduce the memory bandwidth required to compute attention across context windows spanning hundreds of thousands or millions of tokens.

To continue increasing model capacity without incurring a proportional rise in computational cost for every generated token, many large language models replace standard dense feed-forward networks with a Mixture of Experts (MoE) architecture. Pioneered in models such as Google's Switch Transformer and GShard, an MoE layer divides the feed-forward network into numerous specialized sub-networks, or "experts," controlled by a learned routing mechanism. During processing, the router selects only a small subset of experts to activate for each individual token. This sparse conditional computation allows a model to store vast knowledge across hundreds of billions of total parameters while keeping inference speed and energy expenditure comparable to a much smaller dense model.

Constructing a large language model occurs in distinct phases, beginning with self-supervised pre-training. During pre-training, the model's parameters are initialized randomly and exposed to trillions of tokens collected from web crawls, digitized books, academic repositories, encyclopedias, and source code.²⁰ The training pipeline must extensively filter this raw corpus to remove duplicate documents, spam, personally identifiable information, and toxic material.²¹ As the model processes batches of text across thousands of interconnected accelerators using distributed frameworks like Megatron-LM, ZeRO, or Fully Sharded Data Parallel (FSDP), it repeatedly attempts to predict the next token in a sequence.¹⁰ The difference between its predicted probability distribution and the actual next token is measured via cross-entropy loss, and optimization algorithms adjust the model's weights to minimize this error.²²

The relationship between model size, dataset volume, computational budget, and pre-training cross-entropy loss is governed by empirical power-law relationships known as neural scaling laws.²³²⁴ In 2020, Jared Kaplan and colleagues at OpenAI published scaling laws suggesting that model performance depended primarily on parameter count, leading to the training of massive models like the 175-billion-parameter GPT-3 on roughly 300 billion tokens, a ratio of about 1.7 tokens per parameter.²⁵²⁶ In 2022, Jordan Hoffmann and researchers at DeepMind revised this understanding with the "Chinchilla" scaling laws, demonstrating that earlier models were significantly undertrained relative to their size. For compute-optimal training, the Chinchilla findings showed that model parameters and training tokens should be scaled in equal proportions, requiring roughly 20 tokens per parameter. Because high-quality human-written text on the public internet is finite, developers increasingly supplement pre-training corpora with synthetic data generated and curated by existing models.²⁷

When a trained model generates text—a phase called inference—it operates autoregressively: it processes the user's input prompt, outputs a probability distribution over its entire vocabulary for the next token, selects one token via sampling strategies, appends that token to the input sequence, and repeats the process until a stop condition is met. Because inference requires loading billions of parameters from memory for every generated token, engineers employ optimization techniques to reduce latency and hardware costs. Quantization methods, such as GPTQ and QLoRA, compress model weights from 16-bit floating-point numbers to 8-bit or 4-bit integers with minimal loss in accuracy, while memory management systems like PagedAttention and decoding techniques like speculative decoding accelerate throughput.

User interaction with a model is shaped by prompt engineering and test-time compute scaling. Researchers discovered in 2022 that instructing a model to "think step by step"—a technique termed chain-of-thought prompting—significantly improved accuracy on logic and arithmetic problems by allowing intermediate computations to be represented as explicit tokens in the context window. Reasoning-focused models internalize this mechanism natively. Instead of emitting an immediate response, models such as OpenAI's o1 and o3, DeepSeek-R1, and Anthropic's extended-thinking systems allocate additional test-time compute to generate thousands of internal reasoning tokens, exploring multiple solution paths, backtracking from errors, and verifying intermediate steps before presenting a final answer.

To overcome the limitation that a model's internal knowledge is frozen at the end of its training run, large language models are frequently integrated into external software harnesses. Retrieval-Augmented Generation (RAG) connects an LLM to an external knowledge base or live search index. When a user submits a query, the system splits reference documents into smaller chunks, retrieves the passages most relevant to the query using vector embeddings, and inserts those passages into the model's context window so its response is grounded in verifiable data. Furthermore, modern models are trained for tool use and agency, enabling them to format structured calls to application programming interfaces (APIs), execute code in sandboxed interpreters, query databases, and coordinate multi-model workflows—such as pairing one model for initial synthesis with a second model to audit citations and factual accuracy.

Evaluating the capabilities of large language models requires standardized benchmarks because pre-training loss alone does not reliably predict downstream task accuracy. As models have grown in scale, they have exhibited what researchers term "emergent abilities"—capabilities that remain near random chance in smaller models but appear sharply once parameter counts and training compute cross certain thresholds, though some researchers contend this sharpness is partly an artifact of evaluation metrics.

Academic and industry leaderboards assess models across diverse domains.³⁰ General knowledge and multitask reasoning are commonly tested using Measuring Massive Multitask Language Understanding (MMLU) and comprehensive frameworks such as Stanford's Holistic Evaluation of Language Models (HELM) and Beyond the Imitation Game (BIG-bench). Mathematical problem-solving is evaluated on datasets like GSM8K, reading comprehension on DROP, and software engineering proficiency on coding benchmarks such as HumanEval. As frontier models saturated many static benchmarks, researchers introduced harder evaluations for abstract reasoning and fluid intelligence, such as the ARC-AGI benchmark, where OpenAI's o3 model achieved an 87.5% score in early 2025, alongside complex deep-research benchmarks like DRACO.

Beyond natural human languages, large language models demonstrate strong proficiency in formal and symbolic systems. Because training corpora include vast repositories of computer source code, models can translate between programming languages, debug software, and autonomously author complex applications through specialized interfaces such as GitHub Copilot and Claude Code. Their pattern-recognition capabilities have also been applied to non-linguistic sequences, including mathematical equations, chemical signaling, musical notation, and genetic sequences.