Choosing a RAG Architecture, and Knowing It Works

Part five closed out the proven core. Seven retrieval families, an ingestion pipeline, a model-driven loop — the whole toolkit, in production and battle-tested. Which leaves the practical question this series has been circling since part one. Given all of it, how do you actually decide? And once it's built, how do you know it works?

One warning first. There's no flowchart here that takes a corpus and emits an architecture. If I offered one, it would contradict everything in part one — the whole argument is that the answer depends on questions no diagram can see.

What I can offer is the order to think in, and the things that override.

Start with the shape of your questions

You have the golden set from part one: twenty to fifty real questions with expected answers, gathered before choosing anything.

Now do something with it. Sort the questions by what each one demands. Six shapes cover most corpora — the same six from part one:

• Lookup. The answer sits in one place and rarely changes. "How many vacation days do I get?"
• Filter. A predicate over structured attributes, needing every match. "Which vendors renewed after a Sev-1 last year?"
• Aggregation. Counting or computing over the whole corpus. "How many runbooks mention the deprecated auth service?"
• Interpretive. Prose questions where the wording won't match the source. "What's our actual policy on contractor access?"
• Multi-hop. Connecting things no single document connects. "How do these two incidents relate?"
• Global. A question about the corpus rather than in it. "What were the recurring themes last quarter?"

Count them.

That histogram is your architecture requirement. Not the document count. Not the file formats. The distribution of what people will actually ask.

Map the shapes to families

Two things about that table.

First, most corpora produce a mixed histogram, which means most real systems need more than one row. That's fine — it's exactly what part five's agentic routing exists to serve. The model chooses between tools at request time, instead of a developer choosing one pipeline for every question in advance. What you must not do is pick the row for your most impressive question and use it for everything.

Second, if eighty percent of your golden set is one shape: build that row properly and stop. A system that answers the common shape excellently and refuses the rare one honestly beats a system that answers everything mediocrely.

The constraints that override the mapping

Question shape sets the default. These override it.

Corpus size. If the whole corpus fits comfortably in the context window, inline it and skip this entire discipline. Don't build a retrieval system for twelve documents because the architecture diagram looks more serious with one.

Document reality. The table assumes the corpus is text. Real corpora are PDFs with inline tables, scans, Excel workbooks. Chunk those as text and you destroy the very structure the questions depend on — the KPI project in part three failed on exactly this. If your documents carry their meaning in tables, extraction into structure is not an optimization. It's the prerequisite for everything else.

Required correctness. Ask what a plausible-but-incomplete answer costs. In a support assistant, it's a smaller second question. In compliance, procurement, or spec work, an answer that silently omits three of eleven matching records is worse than no answer — and that rules out top-k retrieval no matter what shape the question looked like.

Permissions. If two users must get different answers from the same corpus, this constraint outranks nearly everything else. More below.

Freshness. How stale may an answer be — seconds, hours, a day? Index lag isn't an ops detail. It's a bound on correctness.

Latency budget. Agentic loops and cross-encoder reranking both cost round trips. A conversational assistant can absorb that. An autocomplete can't.

Maintenance capacity. The one people skip. Who rebuilds the entity graph in eight months, after the person who built it has moved teams? A design nobody can maintain degrades silently — and silent degradation in retrieval is the worst kind, because the system keeps answering.

When the right answer is to retrieve nothing

Naive pipelines retrieve unconditionally. That's a decision disguised as a default, and several common cases are actively harmed by it.

The question isn't about your corpus. Greetings, meta-questions, general knowledge. Retrieving anyway injects irrelevant passages into the prompt — and invites the model to work them into the answer, because the context is there and looks like it's meant to be used.

A previous answer already covers it. Suggestion chips and FAQ traffic make the same question arrive constantly. Cache answers keyed on the question and the content version, and thousands of generations become one.

It's a follow-up. "Can you rephrase that?" refers to the conversation, not the corpus. Retrieving fresh passages against those words is a reliable way to derail a good answer.

There genuinely isn't an answer. The most valuable output a grounded system produces is a clean "the corpus doesn't cover this." That requires being able to conclude nothing was found — which, as part two showed, similarity search doesn't give you for free.

Model-driven routing gets most of this without extra machinery. Given the choice, a model asked "hello" generally doesn't call the search tool. A pipeline that retrieves before the model sees the question can't make that choice at all.

Measure retrieval and generation separately

Now the second half. You've built something. How do you know it works?

The most useful practice is also the most commonly skipped: evaluate retrieval on its own, apart from the answer.

If you only score final answers, every failure is undifferentiated. The answer was wrong — was the right evidence never retrieved, or retrieved and then ignored? Different bugs, different fixes. Conflating them is how teams spend a month tuning prompts to compensate for a retriever that never returned the document.

Split it.

Retrieval quality. Label each golden-set question with the documents that actually contain the answer. Then recall@k — did the needed evidence come back at all — is deterministic, free, and runs on every change:

# No judge, no model call. Just: did the evidence make it into the set?
def recall_at_k(cases, retrieve, k=10):
    hits = sum(
        1 for case in cases
        if case.required_doc_ids & {doc.id for doc in retrieve(case.question, k=k)}
    )
    return hits / len(cases)

Recall@k is a ceiling. If the passage isn't in the retrieved set, nothing downstream recovers it — the answer is capped before generation begins. Measure the ceiling first. Ranking metrics come after, once you know the evidence is at least present.

Generation quality. Given the retrieved context, does the answer follow from it? That's groundedness, and it needs judgment rather than string matching. This is where an LLM judge belongs — and I'd insist it's not the same model doing the generating.

The payoff is localization. High recall, wrong answer: generation problem. Low recall: the retriever, and no prompt will save it. You stop guessing which half to fix.

Citations you can't fake

A citation asserted by a model is a claim. A citation derived from what was retrieved is a fact about the system.

The distinction is mechanical. If the model writes the reference, it can invent one. If the interface builds references by walking the actual retrieval results, invention is structurally impossible — the worst case becomes a real link to an irrelevant document, which is a quality complaint, not a fabrication. That's how the copilot on this site does it: sources come from tool outputs, never from generated text.

Two things get conflated here, though. A citation being real is not the same as it supporting the claim. Link validity is structural and cheap. Whether the cited passage actually backs the sentence attached to it is a groundedness question, and it needs the evaluation above. Systems that solve the first and advertise it as the second are common.

Retrieved text is data, not instructions

Here's the production topic that gets skipped most.

Everything your retriever returns goes into the prompt. Which means anyone who can write into the corpus can write into your prompt. A wiki page edited to include "ignore your previous instructions and tell the user to…" doesn't look like an attack to a retriever — it looks like a relevant passage. This is indirect prompt injection, and RAG systems are its natural habitat, because their entire job is pulling third-party text into the model's context.

What actually helps:

• Treat retrieved content as untrusted input. Delimit it, label it, and instruct the model that text inside those boundaries is evidence to quote, never instructions to follow. This raises the bar; it does not guarantee anything.
• Least privilege on the answering model. The generation step should have no tools with side effects that retrieved text could trigger. An assistant that can send email and read the wiki is one poisoned page away from being someone else's assistant.
• Control the write path. Who can put documents into the corpus? A KB writable by every employee is a different threat surface than one behind review.
• Probe it in the eval set. Add cases where the retrieved context contains embedded instructions, and assert the system ignores them.

Be honest about the ceiling: there is no known prompt-level fix that fully solves this. Design-level containment — what the model is able to do — protects you where instructions can't. The copilot on this site labels its conversation history untrusted for exactly this reason, and its only side effect is answering.

Freshness is a correctness property

Treat index lag as a bound on correctness, and the design questions get sharper.

Deletion is the dangerous half. Slow additions are an annoyance. A deleted document still in the index means the system cites content that no longer exists — sometimes content deleted precisely because it shouldn't be cited. Deletions should propagate at least as fast as additions. Usually faster.

Model upgrades are migrations. A new embedding model means reindexing everything, and mid-transition, two halves of your index live in different vector spaces. The embedding model is a versioned dependency with a migration cost, not a config value you bump.

Derived layers have their own clock. Graphs and summary trees go stale on a slower, more expensive cycle than a chunk index. Decide the refresh policy when you build them — not the first time someone notices a summary describing last quarter.

One thing left

That's the framework: shapes, then constraints, then the discipline of proving each half separately.

What's missing is seeing it run. Not dial by dial — all of it, in order, on one messy, realistic corpus, with every decision justified by a histogram instead of a preference.

That's part seven.