RAG Is a Question, Not a Pipeline

The requirement arrives in almost the same words every time: we need the assistant to answer questions from our knowledge base.

Then watch what gets asked next. How many documents? What format — PDF, Confluence, SharePoint? How should we chunk them?

Someone has a vector database comparison open before the first meeting ends.

Here's how that played out on one project I was pulled into. A team had built an assistant over specification documents. The standard pipeline: chunk the specs, embed the chunks, retrieve the top matches, generate. It demoed well. Then the project team started testing it against real questions, and the answers kept coming back wrong in a way nobody could quite name.

We sat down with the failing questions. The diagnosis fit in one sentence.

The questions needed every requirement that applied. The pipeline was built to return the five most similar chunks.

Ask "what are the requirements for X" against a spec, and an answer listing seven requirements out of eleven isn't mostly right. It's wrong, and wrong in the way spec work punishes hardest — the missing four are precisely the ones nobody goes back to check. Top-k retrieval doesn't risk that failure. It guarantees it. Quietly, by design, on every question of that shape.

Nobody on that team was careless. They optimized what the requirement named. The requirement said "knowledge base," so the work became documents, formats, chunking.

But the requirement was never really about the knowledge base.

It was about the questions.

The confusion is earned

Before criticizing the default, be fair about where it came from.

Say RAG and most people see one diagram: documents → chunks → embeddings → vector store → similarity search → top-k into the prompt → generate. The literature calls this naive RAG now. The name is descriptive, not an insult.

The reason that diagram feels like the definition is that it very nearly was. The paper that named RAG — Lewis et al., 2020 — didn't describe a general pattern. It described a specific architecture: a dense passage retriever wired to a seq2seq generator, trained end to end. Dense retrieval wasn't one option among several. It was the thing.

Then the field generalized the term to the pattern — retrieve something, ground generation on it — while a lot of practitioners kept the original implementation in their heads.

The word broadened. The mental image didn't.

So when a teammate says "we'll use RAG" and means "we'll embed the documents," they're not confused. They're using the word the way it was coined. The field moved; the vocabulary didn't move with it.

What RAG actually is

Stripped to the load-bearing idea:

Retrieval-augmented generation means fetching context at inference time so the answer is grounded in your data instead of the model's weights.

That's the whole approach.

Now notice what the definition doesn't say. It doesn't say the context is found by cosine similarity. It doesn't say the corpus is chunked. It doesn't say retrieval happens exactly once, or before generation, or at all for any particular request.

Every one of those is a decision. In naive RAG, they all get made at the same moment, by the diagram, silently.

Five decisions hiding in one diagram

Pull the diagram apart and RAG decomposes into five independent choices. Naive RAG is one setting of each dial. A reasonable default that got mistaken for the machine.

1. Substrate — what are you retrieving from? Raw documents are the assumption. But the substrate can be structured records, an API, a relational database, a knowledge graph, prior conversation, or a layer you build on purpose: summaries, extracted entities, distilled cards. The substrate is a design decision, not an inheritance from whatever happened to be in SharePoint.

2. Selection — how do you decide what's relevant? Dense vectors are one mechanism. Lexical search is another, and it still beats embeddings on exact identifiers, error strings, and rare proper nouns. A deterministic query is a third: a SQL filter, an API call, a typed function. Exact, not approximate. Graph traversal is a fourth. And there's always the option of not selecting at all — if the corpus fits in the context window, retrieval is the trivial function that returns everything.

3. Routing — when do you decide to retrieve? Naive RAG retrieves once, always, before generating. But retrieval can be skipped when it isn't needed. It can be chosen by the model as a tool call. It can be repeated — the model reads what came back, notices it isn't enough, and asks again. Routing is where the biggest quality differences live, and it's the dial teams most often don't realize they've set.

4. Granularity — what actually enters the prompt? Fixed-size chunks with overlap are a convention, not a law. The unit can be a sentence window, a whole document, a parent section retrieved through a matching child, an entire entity record. Chunking decisions routinely matter more than the choice of vector database. That is not the ratio of attention they get.

5. Verification — how do you know the answer came from the data? Are citations derived from what was retrieved, or asserted by the model? Does retrieval respect the asker's permissions? Can you measure grounding at all? This dimension gets skipped entirely, then discovered in production.

A vector database answers one of these five questions. It's the selection dial, set to "approximate semantic similarity." An excellent setting for a real class of problems.

It is not an architecture.

One knowledge base, six questions

Here's the argument at its most compressed.

Take one corpus. The internal documentation of a mid-size company: policies, specs, runbooks, incident write-ups, vendor contracts, meeting notes. Now listen to what people actually ask it.

"How many vacation days do I get?" A lookup. One paragraph of one document, rarely changes. Similarity search finds it. So would a keyword match. A cached answer beats both on latency and cost — and the real engineering question is why you're paying for a generation at all on the ten-thousandth identical ask.

"Which vendors renewed after a Sev-1 incident last year?" A filter and a join over structured attributes. There is a correct, complete answer, and similarity search cannot produce it. Not because it's tuned badly — because top-k nearest neighbours is the wrong operation. Ask for vendors, get the most vendor-ish passages, silently truncated at k. This needs a query, not a search.

"What's our actual policy on contractor access?" Interpretive. The answer exists in prose, but nobody wrote a sentence with those exact words in it. The asker's vocabulary won't match the document's. This is the question dense retrieval was born for.

"How do these two incident reports relate?" Multi-hop. The connection may live in neither document — a shared root cause named in a third. One retrieval pass over the question text won't surface it, because the connecting evidence doesn't resemble the question. This needs a graph, or an agent that can retrieve, read, form a hypothesis, and retrieve again.

"How many runbooks mention the deprecated auth service?" An aggregation. Retrieval returns top k; the answer needs all n. A similarity-based system answers this confidently and wrongly, reporting on the handful it fetched as if that were the population. This needs enumeration or code, not retrieval. Same family as the spec assistant that opened this piece.

"What is this corpus actually about?" Global. No chunk contains the answer, so chunk retrieval structurally can't find it. This needs a summary layer built ahead of time, above the documents.

Same corpus. Six questions. Six different right answers, of which exactly one is naive RAG.

And that's what I keep arriving at from the other direction, in rooms where the system already exists and underperforms: the retrieval strategy is a property of the question, not of the knowledge base. The knowledge base tells you what's answerable. The questions tell you what to build.

Start with the golden set

Which brings me to the practice. The one thing here worth adopting immediately.

Before choosing an architecture, ask for a golden set: twenty to fifty real questions, written by the people who will actually use the system, each paired with the answer they'd consider correct.

Not sample questions invented by the project team. Real ones. From the support queue, the internal chat, the ticket history, the "hey, quick question" messages.

Asking for this is usually met with mild impatience. It looks like a delay before the real work starts.

It is the real work. Three reasons.

It reveals the distribution of question shapes. Sort the set by what each question demands — lookup, filter, aggregation, interpretive, multi-hop, global. That histogram is your architecture requirement. Eighty percent paraphrase-tolerant lookups? Naive RAG is genuinely the right call, and now you know it instead of hoping it. A third filters and aggregations? You know before writing any code that a vector store alone will fail.

It surfaces the questions nobody scoped. The set almost always contains a few questions that quietly assume capabilities nobody planned for: comparisons across time, ownership lookups, exhaustive listings. Finding those in a spreadsheet is cheap. Finding them in production, a week after the stakeholder demo, is not.

It becomes the evaluation set. The list you used to choose the architecture is the list you grade it against afterwards. The eval comes free, and it comes from the only distribution that matters — what people actually ask. Without it, "is it working?" degenerates into someone senior trying five questions they invented on the spot.

There's an asymmetry worth naming here. A document inventory feels like progress — counts, diagrams, artifacts — and tells you almost nothing about which retrieval you need. A question inventory feels like delay and tells you almost everything.

I hold the copilot on this site to the same rule. Its eval cases are questions with expected behavior, graded against the corpus, and the suite grows from what visitors actually ask — not from what I imagined worth testing.

What's coming

This is the first of eight parts, and they follow the five dimensions. Each part deepens one of the dials above — which is the whole argument, turned into a table of contents.

Part 2 — Four ways to retrieve text. Selection. Inlining, lexical search, dense vectors, and hybrid with reranking — the families most teams choose between. What each is good at, how each fails, and an honest defence of dense retrieval where it wins outright.

Part 3 — When retrieval isn't search. Selection, continued. Structured queries, graph traversal, and hierarchical summaries: the three families that abandon similarity altogether, and that fix the failures the first four can't.

Part 4 — Before you retrieve. Substrate and granularity. Parsing, tables, images, chunking, metadata — the ingestion pipeline that decides most of your quality before the first embedding, and gets the least attention.

Part 5 — Agentic RAG. Routing. Letting the model decide when to retrieve, which tool to use, and whether what came back was good enough — the highest-leverage change in the discipline, and its named patterns.

Part 6 — Choosing an architecture, and knowing it works. Verification. The decision framework, the cases where the right answer is to retrieve nothing at all, and the production half: citations you can't fake, permissions, freshness, injection, and measuring retrieval separately from generation.

Part 7 — RAG, end to end. One realistic corpus taken through the whole method: golden set, histogram, substrate work, architecture, evaluation. Every decision from the series made once, in order, on the same problem.

Part 8 — The RAG frontier. Where the proven core stops and the bets begin: recursive language models that compute over the corpus in code, and Google's Open Knowledge Format standardizing the substrate. Potential, limits, and an honest snapshot.

If you take one thing from this part, make it the golden set. The RAG projects I've seen struggle mostly didn't struggle because someone picked the wrong database. They struggled because nobody wrote down the questions first.