Retrieval-augmented generation is the technique of finding the relevant parts of your own data and giving them to a language model at question time, so the answer comes from your information rather than the model's memory. It is how you get an AI assistant that knows your policies, your documentation, and your customer history without training anything.
Almost every business problem described as "the AI does not know our information" is a retrieval problem, and retrieval is cheaper to build, cheaper to update, and easier to audit than fine-tuning. This guide covers how RAG works, the seven steps to build one, what it costs, and the failure modes. The core message, if you read no further: RAG systems fail on retrieval quality far more often than on model quality, and almost everyone spends their effort on the wrong half.
How RAG works, in five steps
- Ingest. Your documents, tickets, database rows, or wiki pages are collected and normalised into text.
- Chunk. Each document is split into passages, typically 200 to 800 tokens, because you retrieve passages rather than whole files.
- Embed and index. Each chunk becomes a vector capturing its meaning, stored in a vector database alongside keyword and metadata indexes.
- Retrieve. At question time, the query is embedded, the closest chunks are found, and results are re-ranked for relevance.
- Generate. The top chunks are passed to the model with the question and an instruction to answer only from the provided context and to cite sources.
Step 5 is where people focus. Steps 2 through 4 are where systems succeed or fail. If retrieval hands the model the wrong three paragraphs, no model produces the right answer.
RAG vs fine-tuning vs long context
| RAG | Fine-tuning | Long context | |
|---|---|---|---|
| Teaches the model | Facts | Style, format, task behaviour | Facts, temporarily |
| Update cost when data changes | Minutes, re-index | Full retraining run | None, but re-sent every call |
| Can cite sources | Yes | No | Partly |
| Data volume ceiling | Effectively unlimited | Limited by training set | Hard context limit |
| Per-query cost | Low to medium | Lowest | Highest |
| Build cost | $35k–$120k | $50k–$200k | $10k–$30k |
| Auditability | High | Low | Medium |
Choose RAG when facts change or must be cited. Choose fine-tuning when you need consistent output format or task behaviour at high volume, generally above roughly 100,000 calls a month. Choose long context only for genuinely small, stable corpora, because stuffing 100 pages into every request is the most expensive way to solve this and it degrades accuracy as context grows.
Auditability is the underrated column. A RAG answer can show which document it came from, which is what makes it defensible to a customer, an auditor, or a regulator. That matters directly under the EU AI Act transparency obligations.
The seven build steps
Step 1: Define correct before you build anything
Collect 50 to 200 real questions with expected answers and the document each answer should come from. This is your evaluation set and it is the single highest-leverage artefact in the project. Without it you cannot tell whether a change helped. Teams that skip this iterate by vibes for three months.
Step 2: Fix the source data
Retrieval quality is bounded by source quality. Deduplicate. Remove superseded documents, because a model cannot know your 2023 policy was replaced. Strip navigation chrome from scraped pages. Convert tables and PDFs properly, since a badly extracted table becomes unusable noise. This step is routinely underestimated and often matches the cost of everything else.
Step 3: Chunk with structure, not by character count
Naive fixed-size chunking splits mid-sentence and mid-table, destroying meaning. Split on document structure: headings, sections, list boundaries. Keep 10% to 20% overlap so context is not lost at the seam. Attach metadata to every chunk: source, section title, date, access level. That metadata does most of the work in step 5.
Step 4: Use hybrid search, not vectors alone
Pure vector search fails on exact identifiers: product codes, error numbers, names, policy references. Combine semantic vector search with keyword search and merge the results. This single change is the largest accuracy improvement available in most systems, and it is the most commonly missing piece we find when auditing an underperforming RAG build.
Step 5: Re-rank and filter by permission
Retrieve 20 to 50 candidates, then re-rank to the best 3 to 8 with a cross-encoder. Retrieving more and re-ranking down beats retrieving few, on both accuracy and cost, because fewer final chunks means fewer tokens.
Apply permission filters at retrieval time, not in the prompt. Never retrieve a chunk the user is not allowed to see and rely on the model to withhold it. Models can be talked out of instructions; a database filter cannot. This is the most serious security mistake in RAG systems and we find it regularly.
Step 6: Constrain generation
Instruct the model to answer only from provided context, to cite the source of each claim, and to say it does not know when the context is insufficient. That last behaviour is a feature and you should measure it: a system that never says "I do not know" is a system that is making things up.
Step 7: Instrument everything
Log the query, the retrieved chunks, the final answer, latency, and token cost. Run the evaluation set on every change. Track the refusal rate and the citation rate. Without this you cannot debug a wrong answer, because you cannot tell whether retrieval or generation failed.
What it costs
| Scope | Build | Timeline | Monthly run |
|---|---|---|---|
| Single source, internal use, under 5k documents | $18k–$35k | 4 to 6 wks | $300–$1,200 |
| Multi-source, customer-facing, permissions | $45k–$90k | 8 to 14 wks | $1,200–$5,000 |
| Enterprise, SSO, residency, audit trail | $90k–$200k | 4 to 7 mo | $4,000–$15,000 |
Typical build cost distribution: data preparation 30%, retrieval pipeline 25%, evaluation harness 15%, interface 15%, security and permissions 15%. Note how little of that is model work. Full cost context is in our AI agent development cost guide.
The six failure modes
| Symptom | Actual cause | Fix |
|---|---|---|
| Confidently wrong answers | Retrieval returned irrelevant chunks | Hybrid search, re-ranking, better chunking |
| Cannot find exact codes or IDs | Vector-only search | Add keyword search |
| Cites outdated policy | Superseded documents still indexed | Date metadata, recency filter, delete on re-index |
| Users see data they should not | Permissions applied in the prompt | Filter at retrieval, before generation |
| Costs spiral | Too many chunks per query, no caching | Re-rank down, cache repeats, cap per tenant |
| Quality drops after launch | New content, no re-indexing | Scheduled re-index, monitor coverage |
Five of the six are retrieval problems. That is the pattern, and it is why swapping to a better model rarely fixes a struggling RAG system.
What this looks like shipped
The retrieval-plus-evaluation discipline is what made Forge AI inside TracefyHR work in production. It generates real HR features, complete with schema, forms, and approval flows, from a plain-English description, and it runs across 50+ companies managing 2,000+ employees with a 15 minute average setup. The generator was not the hard part. Grounding it in each tenant's actual data model, with permissions enforced at retrieval, was.
Frequently asked questions
What is RAG in simple terms?
Retrieval-augmented generation finds the most relevant passages from your own documents and gives them to a language model along with the user's question, so the answer is based on your information rather than the model's general training. It is the difference between an AI that knows things in general and one that knows your business specifically.
Is RAG better than fine-tuning?
For facts, yes, in almost every case. RAG updates in minutes when data changes, can cite its sources, and handles unlimited data volume. Fine-tuning teaches style, format, and task behaviour rather than facts, and it requires a full retraining run whenever information changes. Most teams who think they need fine-tuning need retrieval.
How much does a RAG system cost to build?
$18,000 to $35,000 for a single-source internal assistant over roughly 4 to 6 weeks, $45,000 to $90,000 for a customer-facing multi-source system with permissions, and $90,000 to $200,000 for an enterprise deployment with SSO, data residency, and audit trails. Running costs range from $300 to $15,000 a month depending on volume.
Why does my RAG system give wrong answers?
Almost always because retrieval returned the wrong passages, not because the model is weak. The three most common causes are vector-only search that misses exact identifiers, fixed-size chunking that splits content mid-thought, and outdated documents still sitting in the index. Adding keyword search alongside vector search and re-ranking results is usually the largest single improvement available.
How do we stop RAG from leaking data between users?
Apply permission filters at retrieval time so a chunk the user is not authorised to see is never fetched. Do not retrieve everything and instruct the model to withhold restricted content, because prompt-level restrictions can be circumvented and database-level filters cannot. This is the most serious and most common RAG security flaw.
How long does it take to build a RAG system?
Four to six weeks for a focused internal assistant on a single clean data source, and eight to fourteen weeks for a customer-facing system with multiple sources and permissions. The variable that most affects the timeline is data quality: cleaning, deduplicating, and correctly extracting source documents regularly takes longer than building the pipeline itself.
Key takeaways
- RAG grounds a model in your data at question time. No training required, and answers can cite sources.
- Five of the six common failure modes are retrieval problems, not model problems.
- Hybrid search plus re-ranking is the single largest accuracy improvement in most systems.
- Enforce permissions at retrieval, never in the prompt.
- Build the evaluation set first. Without it you are iterating on impressions.
If you are scoping one
The two decisions that determine whether a RAG project succeeds are made in week one: what the evaluation set looks like, and how the source data gets cleaned. Both are cheap to get right and expensive to retrofit. Tell us what data you want the system to know and we will send back a scoping note covering the chunking and retrieval approach, an honest assessment of your data's readiness, and a cost range. If your corpus is small and stable enough that you do not need RAG at all, we will say so. Related: AI development services buyer's guide, adding AI chat to your app, what generative AI is, and our AI development services.