On this page
Definition
A transformer is a neural network architecture built around self-attention, which enables efficient modelling of long-range dependencies in sequences.
Simple explanation
The transformer's trick is that when it reads a sentence, it can pay different amounts of attention to different words. That lets it capture context in ways older architectures could not.
Why it matters
Introduced in 2017's 'Attention is All You Need', transformers underpin GPT, Claude, Gemini, Llama and almost every modern generative AI system.
How it works
- 1TokeniseInput text is split into tokens.
- 2AttentionEach token attends to all others to build a context-aware representation.
- 3LayersMany attention layers stack, refining understanding.
- 4PredictThe final layer predicts the next token.
Real examples
Products named for illustration only. Inclusion is not an endorsement.
- GPT familyOpenAI's flagship transformer models.
- Claude familyAnthropic's transformer-based models.
- Llama familyMeta's open-weight transformers.
Advantages
- Parallelisable, so training scales.
- Handles long-range context.
- Extends beyond text to images, code and audio.
Limitations
- Compute-hungry.
- Quadratic cost with sequence length (mitigated in modern variants).
Common misunderstandings
- ClaimTransformers understand language.RealityThey model statistical structure of language extremely well.
Frequently asked questions
Who invented the transformer?
Vaswani et al., in a 2017 paper at Google.
Are all LLMs transformers?
Nearly all leading ones are, though alternative architectures exist.
The Tool Money Lab perspective
The transformer is one of the most important computer science ideas of the last decade. Even if it is eventually replaced, the family of ideas it introduced — attention, scale, self-supervision — will shape AI for years.
Conclusion
Transformers are the shape of modern AI. Understanding them is optional to use AI, but valuable if you want to reason about what it can and cannot do.