When Retrieval Isn't Search: Queries, Graphs, and Summary Trees

Part two covered the four families most teams choose between — inlining, lexical, dense, hybrid — and ended on the assumption they share: retrieval means finding the text most similar to the question.

The three families here don't share it. One replaces similarity with an exact query. One replaces it with traversal over a structure you build. One gives up on finding the answer at all, and writes it in advance.

They're less popular. They're also where the failures from part two get fixed.

Structured queries and tools

Here, retrieval isn't search at all. It's a function call with typed parameters — a SQL query, an API request, a filter over metadata. The model's job is translating intent into arguments, not judging relevance.

The shift is from ranking to answering. A similarity search over vendor documents returns the passages that look most vendor-ish and stops at k. A query returns every row matching the predicate, in order, with a count.

One is a guess with good manners. The other is an answer.

A second project made this vivid for me. The ask was an assistant that could answer KPI questions from a pile of Excel workbooks and PDF reports with tables inline. The first attempt was the standard pipeline — embed everything, retrieve, generate.

It couldn't do any of it. Not poorly. At all.

Chunk a spreadsheet and you get fragments of rows torn from their headers. Embed those, and similarity search returns shreds of tables that look vaguely related to the question — from which the model then does arithmetic, confidently. Every number it produced was a guess wearing a citation.

The fix had nothing to do with retrieval tuning. Extract the tables into actual structure once, at ingestion. After that, a KPI question becomes what it always was: a computation. The assistant's job is translating the question into it.

It was never a retrieval problem.

Good at: exactness and completeness — precisely where similarity gives up. Counts, filters, sorts, joins, date ranges. Permissions come along for free, because the query executes as the user instead of reading from a pre-embedded blob that has no idea who's asking. So does auditability: you can log exactly what was fetched, and citations built from returned rows can't be fabricated.

Breaks when: the structure doesn't exist — someone has to build it, which is real work and the subject of part four. And when schemas get large, generated SQL degrades: the model produces queries that are syntactically valid and semantically wrong, which is worse than an obvious error because nothing announces it. It also can't touch genuinely unstructured questions. "What tone does this report take" is not a query.

Reach for it when: the question is a database query wearing a natural-language costume. A large share of "our RAG can't answer this" complaints are exactly that — questions that were never retrieval problems in the first place.

One more thing. Where you can enumerate the question shapes, a handful of constrained, typed tools beats free-form text-to-SQL. Validated parameters give the model far less room to be creatively wrong. That's how the copilot on this site works: a small set of typed filters over the content, not a model composing queries against a schema.

Graph retrieval

Graph retrieval changes the question. Not which text is relevant? but what is connected to what? — and answering that requires building something the corpus doesn't ship with.

How the graph gets built. An extraction pass runs over the corpus, usually with a model. For each chunk: pull out the entities it mentions (systems, people, vendors, incidents, clauses) and the relationships between them — depends on, caused by, owned by, supersedes — every edge keeping a pointer back to its source text.

Then comes the step that decides whether any of it works: entity resolution. The "Auth Service" in a runbook, "authentication-service" in a deploy log, and "AuthSvc" in a postmortem have to collapse into one node. If they don't, the graph quietly fragments into synonyms, and traversal finds nothing.

How retrieval works. Two modes, for two kinds of question.

Local traversal starts from the entities named in the question and walks outward a bounded number of hops, collecting neighbours and their source text. The retrieved set is no longer "text resembling the query." It's "everything connected to the things the query is about." That's exactly why multi-hop works — the connecting document never had to resemble the question. It only had to be attached to something that did.

Community summarization goes the other way. Partition the graph into densely connected clusters, summarize each ahead of time, then cluster and summarize the summaries. Questions about the corpus as a whole get answered from community summaries instead of individual passages. This is the pattern Microsoft's GraphRAG popularized.

Take the incident question from part one: how do these two incident reports relate? Similarity search returns both reports, ranked highly, connected to nothing. A graph answers differently:

incident-2043 ──mentions──▶ [ auth-service ] ◀──mentions── incident-2101
                                   │
                             affected-by
                                   ▼
                          change-4471 (deploy, Mar 12)

Neither report mentions the other. The deploy record names neither. But all three touch one node, so a two-hop traversal surfaces the change record as the link between them. No amount of top-k over the question text produces that answer.

Good at: multi-hop questions, relationships and dependencies, lineage and causality — and through community summaries, global questions that chunk retrieval structurally can't reach.

Breaks when: extraction is itself a model pipeline with an error rate, and every downstream answer inherits it silently. Entity resolution is where most of the damage happens. Construction costs at least one model pass over the whole corpus, and the bill recurs — incremental updates are harder than they look, because one new document can merge two entities that were previously distinct, which changes traversal results elsewhere. And for most corpora, it's a lot of machinery standing ready for questions nobody asks.

Reach for it when: the domain is genuinely relational and those questions are frequent. Investigations, compliance lineage, dependency maps, incident causality. Check the golden set first. If multi-hop questions are three out of fifty, a graph is not your answer — a structured query over metadata probably is.

Hierarchical summaries

Some questions have no answer anywhere in the corpus.

"What were the recurring themes in last quarter's incidents" is not written in any incident report. It exists only in the aggregate. No retrieval strategy can find text that was never written.

So you write it in advance.

How the tree gets built. Chunk and embed as usual. Then cluster related chunks and summarize each cluster with a model. Treat those summaries as new documents; cluster and summarize them; repeat. You end up with a tree — raw passages at the leaves, increasingly abstract syntheses toward the root. This is the structure RAPTOR formalized.

How retrieval works. Either descend the tree — start at the top summaries, pick the branch, drill to the leaves — or flatten it and search every node at every level at once, so one query can match a specific sentence or a corpus-wide synthesis, whichever is genuinely closer. Flattened is simpler and usually better, because it doesn't force a routing decision before you know whether the question is narrow or broad.

The conceptual move deserves to be said plainly. Summarization here isn't compression for storage. It manufactures answers that don't exist in the source material, and stores them as retrievable objects.

The cheaper version to try first: skip the tree. One summary per document, used purely for routing — match the question against summaries, then run ordinary retrieval inside the winners. You lose corpus-wide synthesis and keep most of the practical benefit, at a fraction of the cost.

Good at: global and thematic questions, and routing across large heterogeneous document sets where "which document" is the real first question.

Breaks when: summaries discard exactly the detail someone eventually asks for. The abstraction that makes them useful is the same abstraction that loses specifics. They go stale the moment sources change — congratulations, you own a regeneration pipeline. And every summary inherits the blind spots of the model that wrote it.

Reach for it when: the corpus is large and the questions are often about it rather than in it.

Notice what the last two share. Both answer questions the raw corpus can't by building a derived layer above it — a graph of resolved entities, or a tree of generated syntheses. That isn't a selection trick. It's a change of substrate, the first of the five dimensions from part one. The KPI project earlier in this piece is the same move in miniature: tables extracted into structure are a derived layer too.

It's also the most under-used move in practice. Teams tune retrieval for months against a corpus that simply does not contain the answers they're asking for, when the fix was to manufacture the missing layer once, offline.

Two levers that beat the index choice

Before leaving the families, two things that affect quality more than which family you pick — and both get less attention than they deserve.

Granularity. How you cut documents into chunks routinely matters more than the index you put them in. A chunk that severs a conclusion from its premise is individually meaningless, and no ranking algorithm recovers meaning destroyed before indexing. It's enough of a lever that the next part is devoted to it.

Query transformation. The user's question is often not the best search string. There's a whole family of cheap moves that rewrite it before you search — none of which touch the index, all of which are worth knowing exist:

• Rewriting and expansion — clean up a vague question, or pad it with synonyms and related terms so lexical and dense retrieval both catch more.
• Multi-query — generate several phrasings of the question, search each, and fuse the results. RAG-Fusion does exactly this, with the same reciprocal rank fusion from part two.
• Step-back — ask a broader version of the question first, retrieve the background, then answer the specific one; the step-back prompting technique.
• Decomposition — split a compound question into parts and retrieve for each. We'll meet it again, model-driven, in part five.
• HyDE — generate a hypothetical answer and search with that instead of the question, on the bet that a fabricated answer lands closer to the real passage in embedding space than the question does.

None of these is a large project. Any one can move recall more than swapping your vector database would — which is the whole point of this section. The family you pick is rarely where the quality is won.

A summary you can hold in your head

The instruments are ready. The corpus isn't.

None of these seven is a pipeline. They're instruments, and most real systems hold several at once — a structured filter for the part of a question that's really a query, hybrid search for the part that's really prose, an inlined profile for the facts that are always relevant.

But every one of them assumes something we've taken for granted across three parts: that your documents are already in a state a retriever can search. A spec PDF is not. A spreadsheet is not. A scanned contract is not. Before any of these instruments can play a note, someone has to turn a pile of files into a corpus — and that work decides more of your quality than the instrument you finally pick.

That's part four.