What is RAG, and why do RAG pilots stall?
Retrieval-augmented generation (RAG) indexes your documents and lets a language model answer questions with passages retrieved from them, so answers are grounded in your data instead of the model's memory. It is the most common shape of an LLM project, and pilots stall for a predictable reason: a demo takes an afternoon, but then real users arrive with real questions, the index goes stale, answers cite the wrong version of a policy, and trust evaporates. Almost every failure we have seen traces back to one of the twelve checks below being skipped. The technique itself is well established — the original paper showed retrieval plus generation beating a standalone model on knowledge-intensive tasks (Lewis et al., 2020).
RAG vs fine-tuning: which one do you need?
Use RAG when the model needs to answer from documents that change and must be cited; use fine-tuning when you need a consistent style, format or domain vocabulary that prompts cannot enforce. They are not rivals — many production systems fine-tune a small model for format and use RAG for facts — but for "answer questions about our documents" RAG is the default, because it is cheaper to update and every answer carries a source.
| Criterion | RAG | Fine-tuning |
|---|---|---|
| Best for | Facts from changing documents, citations | Style, format, domain vocabulary, narrow tasks |
| Updating knowledge | Re-index the document (minutes) | Retrain (days) and re-evaluate |
| Access control | Per-document ACL at query time | None: everything trained is available to everyone |
| Traceability | Every answer links to a passage | No source; the model "just knows" |
| Upfront effort | Ingestion pipeline, index, evals | Labelled dataset, training runs, evals |
| Typical failure | Bad retrieval, stale index | Overfitting, forgetting, stale facts |
How do you prepare the data?
Give every indexed source an owner and a refresh schedule, preserve document structure during extraction, attach metadata to every chunk, and mirror the source system's access control in the index. Data problems cause more RAG failures than model problems: a chunk that cuts a table in half produces a confident wrong answer, and a document without an access level leaks to the wrong user. OWASP lists vector and embedding weaknesses — retrieval returning documents the user should not see — as a distinct Top 10 risk in its 2025 revision (OWASP, 2025).
- One owner per source. Every indexed collection has a person responsible for its accuracy and a refresh schedule.
- Structure preserved. Tables, headings and lists survive extraction; a chunk never cuts a table in half.
- Metadata on every chunk. Source, date, version, access level — so retrieval can filter and the answer can cite.
- Access control mirrored. If a user cannot open the document, retrieval must not return it. Test this explicitly.
How do you make retrieval reliable?
Combine keyword and vector search, rerank a wide candidate set down to a few passages, measure retrieval quality on its own labelled set, and build an explicit "I don't know" path for when nothing relevant comes back. Retrieval is where most quality is won or lost, and it is cheap to measure: a hundred questions with their correct passages give you a recall number you can track independently of the model. Hybrid search and reranking are the two changes that most often move that number.
- Hybrid search. Keywords plus vectors; product codes and names are found by exact match, concepts by embedding.
- Reranking. Retrieve 30, rerank to 5. Cheap and one of the biggest quality gains.
- Retrieval quality measured on its own. A labelled set of question → correct passages, with recall tracked separately from answer quality.
- "I don't know" path. When nothing relevant is retrieved, the system says so instead of improvising.
How do you keep generation honest?
Require a citation for every claim, replay an evaluation set of real questions on every prompt, model or index change, run output guardrails, and choose the model by numbers on that set rather than by a demo. Confident wrong answers are the failure users remember; OWASP calls it misinformation and ranks it in the Top 10 for LLM applications (OWASP, 2025). NIST's generative AI profile lists confabulation among the risks organisations should measure, and recommends evaluation against representative inputs before deployment (NIST, 2024).
- Citations required. Every claim links to the passage it came from; answers without support are blocked.
- Evaluation set from real questions. 100–300 cases with expected answers, replayed on every prompt, model or index change.
- Guardrails on output. PII, policy and tone checks before the answer leaves the system.
- Model choice by number. Compare two or three models on the evaluation set for quality, latency and cost — not on a demo.
How do you operate RAG after go-live?
Trace every request, feed user corrections back into the evaluation set weekly, alert when a document changes at the source but is not re-indexed, and put cost per answer on a dashboard the business reads. These four operational checks are what turn a pilot into a system people rely on, because they catch drift before users do. They are also where most organisations fall short: McKinsey reports that only a minority of companies have the practices in place to track well-defined KPIs for generative AI solutions (McKinsey, 2025).
- Tracing. Every request stores the query, retrieved chunks, prompt, answer, tokens and latency.
- Feedback loop. Thumbs-down and human corrections flow into the evaluation set weekly.
- Index freshness alerts. A document updated at the source but not re-indexed within its schedule triggers an alert.
- Cost per answer on a dashboard the business reads.
Run all twelve before go-live and RAG becomes boring — which is what you want. Our retail assistant case shows the numbers this produces; to discuss your documents, start with a pilot.