What RAG Actually Is

A customer asks for a chatbot over their knowledge base. The slide deck says "RAG" in the second bullet. You are about to spend six weeks building something that might not need to exist.

This lesson is the one that keeps you from doing that.

What the pipeline actually is

Retrieval-augmented generation is three stages, not a product.

1. Step 1

   Retrieve

   Given the user's query, pull a small set of candidate passages from a larger corpus — usually by vector similarity, BM25, or both.

2. Step 2

   Rank

   Reorder those candidates with a more expensive scorer that reads the query and the passage together, so the top one is the most relevant and not just the first hit.

3. Step 3

   Ground

   Hand the top passages to the model as context, with instructions to answer from them and refer to them by id. The output can now be checked back against its sources.

Nothing about this requires an LLM — information retrieval has been a field for forty years. What the LLM adds is the final step: the answer is written in the user's own words, over the retrieved passages, in one call.

What retrieval buys you

A model with a long context can read a lot. That is a real capability and it is growing on a release cadence, so the argument for retrieval has to survive it. Five things do.

Notice what is not on that list: accuracy. A well-cached prompt against a grounded model will often match a RAG system on raw correctness for a small corpus. The argument for RAG lives in the other four.

The question to ask first

Three customer requests, three different answers.

"Answer questions over this one 40-page policy document."

That is a prompt. Forty pages is about 30,000 tokens, and prompt caching pays the ingest cost once per cache lifetime. No retrieval, no vector store, no chunking strategy. If the customer insists on retrieval anyway because their procurement form has a box for it, say that out loud.

"Answer questions over our 10,000-page product documentation that changes every week."

That is retrieval. Freshness, scale and cost all apply. You will have a vector store, a chunking strategy, and a reranker before you are done.

"Answer questions over the policy document, and also reason about whether a described customer situation complies with it."

That is retrieval and reasoning. The second half is where most RAG systems underperform, because the retrieved passages are relevant but the step from "here are the rules" to "therefore this case is allowed" is a model capability retrieval cannot help with. Say which half is which when you scope it.

Where each stage fails

Each stage has its own failure mode, and separating them is what lets you debug the system when it is wrong.

Retrieval fails when the right passage is not in the top-k. The model never sees it, so the answer is wrong in a particular way: confidently off-topic, or confidently drawn from a nearby but wrong passage. The fix lives in the index — a different embedding model, hybrid search, a smarter chunk.

Ranking fails when the right passage is in the top-k but not at the top. The model sees it but attends more to what came first. The fix is a reranker, or a prompt change that tells the model to weigh the candidates before answering.

Grounding fails when the right passage is in context and the model still makes something up — or stitches two passages into a claim neither one supports. The fix is in the prompt and the judge: say "answer only from the provided passages, cite by id, refuse if the answer is not present", and measure groundedness as its own metric.

When a RAG system is wrong, the first question is which of these three broke. Most teams debug the generation and ignore the retrieval, which is backwards — retrieval is the step that is most often the actual cause.

What to take away

RAG is a three-stage pipeline: retrieve, rank, ground. Retrieval buys you freshness, scale, grounding, cost and an audit trail — not accuracy on its own. Before building one, ask whether the problem is retrieval or reasoning, because a single document plus prompt caching is a cheaper answer than a vector store for a surprising number of customer asks. When a RAG system goes wrong, the three stages fail differently, and knowing which one is wrong is the whole of debugging it.

Next: chunking, which is the quietest source of regression in a RAG system and the one most teams get wrong without noticing.