Retrieval-augmented generation, or RAG, lets an AI system look up relevant material before generating an answer. Instead of asking a chatbot to remember your product manual, you give it passages from that manual to work with. Think open-book exam, with a student who still needs their work checked.
RAG can make answers more useful and easier to inspect. It does not automatically make them correct, current, or private. Here is how the pieces fit together, without requiring a PhD or a cupboard full of expensive acronyms.
Table of Contents
- What is RAG?
- Why use it?
- How it works
- The main components
- RAG versus fine-tuning
- Possible uses
- Common pitfalls
- A practical first experiment
- FAQ
What is RAG?
A RAG system combines a retrieval step with a generation step. It finds information relevant to a question, then supplies that information to a language model as context for its response. The documents remain an external source of information; retrieving them does not itself retrain the model.
Patrick Lewis and colleaguesâ 2020 paper introduced an influential RAG formulation combining a language model with a searchable external memory. The term is now also used more broadly for applications that retrieve material and include it in a modelâs input.
Imagine asking whether a fictional cat carrier is suitable for air travel. A generic answer about carriers is not enough. You want the relevant product dimensions and the particular airlineâs requirements, with their dates. RAG is a way to bring those documents into the answer, rather than hope the model recalls the right combination.
Why use it?
External documents can supply details absent from a modelâs training, including newly published material and information specific to an organization. They can also provide a trail back to the sources. That is useful when you want to ask, âWhere did this answer come from?â
But retrieval is not verification. An obsolete manual can be retrieved perfectly. An answer can cite the right manual while misreading its table. For our imaginary carrier, fetching the airlineâs cargo rules would not necessarily answer a question about taking it into the cabin.
The goal is a system whose reasoning can be checked against appropriate evidence. A confident paragraph with a decorative footnote is still just a confident paragraph.
How it works
A typical document-based setup has two stages: preparing material for search, and answering queries. Preparation needs repeating when documents change; it is not a one-time ceremony followed by permanent freshness.
First, extract readable text, keep the source information, and divide long documents into useful passages. These are often called chunks. An embedding model can represent a passage as a numeric vector, allowing a search system to compare related meanings. Keyword search can also contribute, especially for exact names and identifiers.
Next, retrieve candidates for the question, select useful passages, and give them to the generator. Some systems add a reranker to reorder the candidates before generation. Anthropicâs 2024 contextual retrieval explanation describes combining semantic and keyword retrieval and preserving context around otherwise ambiguous chunks.
In the carrier example, keeping a table heading attached to its measurements matters. A perfectly extracted number with no indication of whether it means height, width, or weight is an unusually efficient way to create a wrong answer.
The main components
- Source preparation: extract content and preserve meaningful boundaries, dates, and document identity.
- Search: find passages relevant to the question, using methods suited to the collection.
- Context selection: remove irrelevant or duplicate material and keep enough evidence to answer.
- Generation and presentation: produce the response and expose its supporting sources.
A vector database is one possible part of the search layer, not the definition of RAG. A small collection may work with a simpler search arrangement. Conversely, a large vector store cannot repair documents that were incomplete before they reached it.
RAG versus fine-tuning
Fine-tuning changes model parameters through further training. RAG supplies external information during answering. They address different needs and can be combined. The 2024 study on RAG, fine-tuning, and an agricultural case study examines these tradeoffs rather than treating one technique as a universal replacement for the other.
For a frequently changing manual, begin by testing whether retrieval supplies the missing facts. If the model repeatedly mishandles a specialized task even with good evidence, improved prompting, training examples, or fine-tuning may be worth investigating. Measure the actual failure before choosing the treatment.
Fine-tuning is not a guarantee of faithful recall, and RAG is not guaranteed to be cheaper. A frequently queried system with long contexts has ongoing inference costs. Training has its own data preparation, evaluation, and compute costs. Compare a defined workload, not two marketing labels.
Possible uses
Useful experiments include a help-center assistant, a search tool for an internal handbook, or a way to ask questions of your own notes. Start with a specific question the collection can actually answer. âWhich steps reset this model?â is easier to assess than âUnderstand everything our company knows.â
For each use, identify who maintains the sources and who may read them. A support answer should not quietly mix public documentation with another customerâs private ticket. A personal notebook can also contain material you do not want sent to an external service.
Common pitfalls
Confusing access with permission
RAG does not automatically keep data on your machine. Hosted embedding services may receive text during indexing, and a hosted generator receives the context included in its request. Storage, retention, logging, and any use for training depend on the services and settings involved.
AWSâs guidance on data authorization emphasizes enforcing access before passing material to a model. Do not retrieve a confidential document and then rely on a prompt asking the model to keep it secret. Apply permissions in the application and retrieval system.
Retrieving instructions disguised as information
A document can contain text trying to redirect the assistant. The 2023 indirect prompt-injection research demonstrates why retrieved content cannot be assumed harmless merely because it arrived through search. Treat source text as data, restrict tool permissions, and test hostile examples. Telling the model to ignore malicious instructions is not a complete security boundary.
Assuming more context must be better
Lost in the Middle found that the position of relevant information affected performance in the models and tasks studied. It is a reason to evaluate context selection, not proof that every model ignores every middle passage.
Try questions that require combining two sections, noticing an exception, or resolving an older version against a newer one. Those are more revealing than repeatedly asking for the sentence that happens to be first in the document.
A practical first experiment
I would begin with a small, non-sensitive document collection and questions whose answers you can inspect manually. Include questions that have no answer in the collection. Otherwise, you never test whether the assistant knows when to stop improvising.
For each response, check three things separately: did search find the needed evidence, did the answer use it faithfully, and did the response address the question? The Ragas evaluation paper describes this kind of multidimensional assessment. Automated scores can help compare versions, but they should not replace examining failures yourself.
For the imaginary carrier, test a missing dimension, a superseded product sheet, and an airline rule that explicitly excludes the route in the question. Also check that a restricted document stays inaccessible. Keep the same questions while changing one part of the system, so improvements are not just impressions. Our discussion of AI coding benchmarks considers another setting where the test you choose matters to the conclusion.
Record enough to investigate failures without turning the logs into a second uncontrolled copy of private documents. Add a process for updates and deletions. The production work is often less glamorous than the demo: somebody has to make sure last yearâs rules stop winning the search.
FAQ
Does RAG require semantic search?
No. Retrieval can use keyword search or other suitable methods. Semantic search is a useful option; generation based on retrieved material is what makes the overall pattern RAG.
How much will it cost?
Estimate document processing, indexing, storage, retrieval, model input and output, and maintenance. Multiply measured per-query costs by your expected volume. For illustration, a one-cent query repeated 10,000 times a day costs about $3,000 over 30 days before other expenses. That is arithmetic, not a quoted provider price.
Does a citation prove the answer?
No. Open the source and check that it supports the specific claim. A link to an airline homepage does not establish a baggage allowance for your flight.
Can an agent use RAG?
Yes. An agent can choose when to retrieve information as part of a larger task. Calling something an agent does not remove the need to check evidence and enforce access.
The cat take: RAG gives the assistant a bookshelf. It still needs to pick the right book, read the right page, and resist confidently explaining the box the book arrived in.
