RAG, End to End: A Worked Example

Frameworks are cheap until you watch one run. So this last part runs the whole series, once, in order, on a single problem.

The scenario is fictional, but assembled from things I've actually seen — readers of part one will recognize the spec assistant, and readers of part three the KPI system. Consider this the composite of the projects that taught me the method.

The setup

A mid-size industrial company. A few hundred people. The ask arrives in the usual words: an assistant that answers questions from our documentation.

The documentation, on inspection:

• Product specification PDFs, several hundred pages each, dense with requirement tables
• Test reports and production metrics in Excel workbooks
• Maintenance runbooks in a wiki
• Incident and nonconformity reports
• Vendor contracts
• Process documentation of varying age and honesty

The team's first instinct is the default: chunk all of it, embed all of it, search, generate. One pipeline, one index, every question equal.

You already know that's where we don't start.

Write the questions down first

We ask for the golden set: real questions from the people who'd use this, each with the answer they'd accept. It takes two weeks of nagging and comes back with forty.

A sample, because the texture matters:

1. "What's the warranty period for the M-series controllers?"
2. "What are all the requirements for enclosure corrosion protection?"
3. "Which suppliers had a nonconformity report open in Q1?"
4. "What was the first-pass yield on line 3 over the last two quarters?"
5. "Does the housing spec allow aluminum for outdoor units?"
6. "Why did the coating process change in March?"
7. "Who approved the last revision of spec 4-112?"
8. "What do our maintenance docs actually cover?"

Eight questions, and already the shapes are pulling in different directions. A lookup. An exhaustive listing. A filter. A computation. An interpretation. A causal chain. A metadata lookup. A global question.

The histogram

Classify all forty by shape — the six from part six — and count:

Shape	Count	Example above
Lookup	13	#1, #7
Interpretive	8	#5
Filter	7	#2, #3
Aggregation	6	#4
Multi-hop	4	#6
Global	2	#8

This table is the architecture. Everything that follows is just reading it.

Start with what it forbids. Thirteen of forty questions — filters and aggregations — need complete answers: every requirement, every supplier, a number computed over all the rows. Top-k retrieval cannot serve a third of the workload, structurally. Question #2 is the spec-assistant failure from part one: an answer listing seven corrosion requirements out of eleven isn't 64% right, it's wrong. And #4 is the KPI failure from part three: the number doesn't exist in any chunk. It has to be computed.

Now what it demands. Twenty-one questions — lookups and interpretive — are classic text retrieval, and eight of those genuinely need semantic matching ("allow aluminum" vs. whatever the spec's actual wording is). That's the real, earned home of hybrid search.

And what it doesn't justify. Four multi-hop questions. Two global ones. Keep that in mind for later.

Substrate before search

Here's the unglamorous conclusion the histogram forces: the biggest engineering effort in this project isn't retrieval. It's extraction.

The corpus carries its most valuable content in structures that chunking destroys. Requirement tables in spec PDFs. Metrics in Excel. So before any index exists, an ingestion pass builds two derived layers:

Requirements become records. Every requirement extracted into {id, spec, section, applies_to, text}. Model-assisted extraction, human spot-checks on a sample, and — because spec IDs and section numbers are verifiable — cheap validation against the source. Now "all requirements for enclosure corrosion protection" is a filter over records. Complete by construction.

Tables become tables. Excel sheets and PDF-embedded tables land in a small analytical store with their headers intact. First-pass yield on line 3 is now a query, not a similarity search over shredded rows.

This is the substrate dimension from part one doing the heavy lifting. Two-thirds of the "hard" questions stopped being hard the moment the data took the right shape.

The instruments

With the substrate in place, the toolset almost writes itself:

• find_requirements(spec?, topic?, applies_to?) — filters over the requirement records. Returns every match and the count.
• query_metrics(metric, scope, period) — computes over the extracted tables. Returns numbers with the rows they came from.
• search_docs(query) — hybrid search with reranking over prose: runbooks, process docs, contract text. Chunks carry contextual descriptions, per part four.
• get_document(id) — full text of one document, for depth.
• list_documents(kind?) — one-line summaries per document, built at ingestion.

Routing is the model's job — part five's argument. A bounded loop, a hard step cap, tool descriptions treated as the routing logic they are. And an answer cache in front of everything, because thirteen of forty questions are lookups and the ten-thousandth "what's the warranty period" should cost nothing.

Citations are built from tool outputs, never model claims. The permission story rides the tools too: queries execute as the asker, so the contract tools simply don't return what the asker can't see.

What we deliberately didn't build

No knowledge graph. Four multi-hop questions out of forty, and each traces changes through records that already reference each other — change logs name NCRs, NCRs name suppliers. The agentic loop can walk those references with the tools it already has. A graph would be a second derived layer, with an extraction pipeline and a maintenance clock, serving 10% of the workload that the loop covers adequately. The golden set says no.

No summary tree. Two global questions. The per-document summaries behind list_documents answer "what do our docs cover" honestly, if not profoundly. If global questions grow, RAPTOR-style synthesis is a known upgrade path. Not day one.

No vector-only pipeline. The thing the team almost built on day zero would have served 21 of 40 questions and confidently failed the rest. That sentence is the whole series.

Proving it

The golden set now becomes the eval, split by what each shape allows:

Deterministic where the answer is exact. Filters and aggregations have correct answers — a set of requirement IDs, a number. Assert them. No judge needed, no ambiguity, and these are precisely the questions where wrong is expensive.

Recall first where retrieval decides. For interpretive questions, label which documents contain the answer and measure recall@k on search_docs before looking at any generated text. It's the ceiling; measure it first.

Judged where wording varies. Grounded prose answers get an LLM judge — a different model than the one generating — scoring whether the answer follows from what was retrieved.

Probed where it can be attacked. A few cases plant instructions inside retrieved content and assert the assistant treats them as text. The wiki is writable by everyone; the eval should remember that.

Then ship thin. The system launches answering four shapes well — 34 of 40 questions — and saying "I can't answer that reliably yet" to the rest. Every question users actually ask gets logged and mined for the next round of cases, so the golden set stops being what we imagined and becomes what's real.

Honest refusal beats confident wrongness everywhere, but especially in a factory.

The proven core, applied

Everything in this build is boring, in the best way. Seven families, an ingestion pass, a model-driven loop, an eval that can fail — all established, all in production somewhere today. What turned that toolkit into a system was the histogram: it fit the tools to this corpus and no other.

That's the thing to carry out of seven parts. There is no general solution for "answers from a knowledge base." A different histogram builds a different system — which is exactly why the method outranks any architecture. The retrieval strategy is a property of the question. The deliverable was never a pipeline; it's a set of instruments, a router to pick between them, and the judgment to know which is which. And that judgment is bought the same cheap way it was in part one — by taking the questions seriously before the tools.

That is the proven core, start to finish.

One thing remains, and it's a different kind of thing entirely. Everything to this point is battle-tested. Part eight is the edge — recursive models that compute over the corpus instead of retrieving from it, and a new standard for the substrate itself. Not what to build today. What to watch.