On this page
Definition
Retrieval-Augmented Generation is a pattern in which relevant documents are retrieved from an external source and injected into an LLM's prompt so the model can answer with grounded, up-to-date, private information.
Simple explanation
An LLM by itself only knows what it was trained on. RAG is how you plug it into your own documents, your product knowledge base, or the live web.
Instead of asking the model 'what does our refund policy say?' and hoping it guesses, you fetch the actual policy first and let the model summarise it.
Why it matters
RAG is what turns an LLM into a real business tool. It makes answers specific, current and defensible — because the source is the model's own retrieved text.
Almost every serious enterprise AI product — customer support, internal search, sales enablement — is a RAG system under the hood.
How it works
- 1IngestSplit documents into chunks and store them.
- 2EmbedConvert each chunk into a vector representing its meaning.
- 3QueryTurn the user's question into a vector too.
- 4RetrieveFind chunks whose vectors are closest to the query.
- 5GenerateSend the top chunks + the question to the LLM as context.
Real examples
Products named for illustration only. Inclusion is not an endorsement.
- PerplexityRetrieves web pages before generating answers with citations.
- ChatGPT File uploadsUses retrieval to answer questions about the files you attach.
- Notion AI Q&ARetrieves from your workspace before generating replies.
Advantages
- Grounds answers in real data — reduces hallucinations.
- Works with private, proprietary or fresh information.
- Cheaper than fine-tuning for most business use cases.
- Answers can cite their sources.
Limitations
- Retrieval quality is the ceiling on answer quality.
- Requires good chunking and embedding choices.
- Adds latency and complexity.
- Sensitive data still passes through the LLM — governance matters.
Common misunderstandings
- ClaimRAG teaches the model your data.RealityThe model does not learn anything — the data is inserted at request time.
- ClaimRAG eliminates hallucination.RealityIt reduces it, especially when the model is instructed to answer only from retrieved context.
Frequently asked questions
Is RAG better than fine-tuning?
For factual Q&A over changing data, usually yes. For teaching a model style or format, fine-tuning is often better.
What is a vector database?
A store optimised for finding items by meaning similarity — the backbone of most RAG systems.
Can I do RAG without a vector database?
Yes — keyword search or even a simple loop works for small corpora. Vectors scale better.
Does RAG make AI safe for enterprise data?
It helps ground answers, but you still need access controls, logging and provider agreements.
The Tool Money Lab perspective
RAG is the least glamorous but most valuable AI technique for businesses. It is how AI becomes useful on your own content instead of the entire internet.
The failure mode we see most is teams trying to fine-tune when they should have retrieved. Start with RAG; only fine-tune when retrieval genuinely cannot solve the problem.
Conclusion
RAG is the pattern that turns generic AI into an expert in your domain. Retrieve first, generate second, cite where possible.
For most 'we want AI over our data' projects, RAG is the answer — and the sooner teams learn to design retrieval well, the better their AI investments perform.