Agentic RAG: Letting the Model Drive Retrieval

Every one of the seven families raised the same question, and I kept deferring it. Given all these instruments — inline, lexical, dense, hybrid, structured, graph, summary — who picks which one runs for a given question?

For most of RAG's short history, the answer was a developer, deciding in advance. This is the routing dimension from part one, and until recently it was set at design time, by rules, before the system ever saw a real question.

The shift this part is about: move that decision into the model, at request time. That's the whole of what "agentic RAG" means, underneath the branding. And it's the highest-leverage change available in the entire discipline.

What actually changes

The standard pipeline retrieves, then generates. One retrieval, decided by a rule, before the model has done any thinking.

Give the model tools and a loop instead, and every one of those decisions becomes the model's:

• Whether to retrieve at all.
• Which tool or index to reach for.
• What query to send it.
• And, crucially — whether what came back was good enough, or whether to try again.

Retrieval stops being a step that precedes generation and becomes a loop the generation drives.

Strip away the vocabulary and agentic RAG is give the model tools and a bounded loop. That plainness is not a criticism. It's why the change is so cheap and so large at once.

Here's the failure it fixes, the one pre-planned routing structurally cannot. A rule-based router decides what to fetch by pattern-matching the question before the model sees it. Phrase your question in a way the rules didn't anticipate and it routes wrong — and there's no recovery, because the model was never given the option to ask for something else. Move the decision into the model, and a bad first retrieval becomes recoverable instead of fatal.

That's the difference between a system that degrades gracefully at its edges and one that falls off a cliff the moment a question is phrased unusually.

The loop has named shapes

"Agentic" gets used as if it were one technique. It isn't — it's a small family of patterns, and knowing them by name is the difference between hand-waving and design. All of them are variations on a single idea, the ReAct loop: the model interleaves reasoning with actions, thinks about what it needs, acts to get it, observes the result, and thinks again.

Within that loop, the useful patterns:

Routing. The simplest. The model looks at the question and picks which tool or source it belongs to — a filter for a filter-shaped question, hybrid search for a prose one. This alone is most of the value, and it's exactly the six-shapes mapping done by the model at runtime instead of by you at design time.

Query decomposition. For compound questions, the model breaks the question into parts, retrieves for each, and composes the answer from the pieces. "Compare the warranty terms of the M-series and the S-series" is two retrievals and a synthesis, not one similarity search that half-answers both.

Active retrieval. Rather than retrieving once up front, retrieve as generation proceeds, whenever the model is about to assert something it isn't sure of. FLARE makes this concrete: it watches the confidence of the text the model is about to produce, and when the next sentence looks shaky, it pauses to retrieve evidence for exactly that sentence before continuing. Retrieval driven by the generation's own uncertainty.

Self-correction. The model — or a cheap dedicated grader — evaluates what retrieval returned before trusting it. Self-RAG trains the model to emit reflection tokens that judge whether a passage is relevant and whether the answer is actually supported by it. Corrective RAG runs a lightweight evaluator over the retrieved set and, when the results look weak, falls back — often to a web search — instead of generating confidently from bad context. Both encode the same instinct: don't build on evidence you haven't checked.

You don't adopt these as a menu. They're points on a spectrum, from "let the model pick a tool" to "let the model plan, retrieve iteratively, and grade its own work." How far along that spectrum you go is a cost decision, which brings us to the part vendors skip.

Single agent, or many

One more axis, because it's where a lot of complexity gets added without being earned.

The patterns above describe one model in one loop with a set of tools. That is enough for the overwhelming majority of RAG systems, and it should be the default.

The heavier alternative is multi-agent: an orchestrator that decomposes a task and hands sub-tasks to specialized sub-agents, each with its own tools and context, whose results get composed back together. It's genuinely useful for broad research-style questions that fan out across many sources — but it multiplies latency, cost, and the number of things that can go wrong, and most "we need multiple agents" instincts are really one agent that needs better tools. Reach for orchestration when a single loop is demonstrably the bottleneck. Not before.

What it costs

The loop is powerful. It is not free, and three costs are load-bearing.

Latency and spend, unbounded by default. Every iteration is another round trip and another bill. And a confused model, left uncapped, will retrieve forever — chasing a good answer it can't find, one expensive step at a time. A hard step cap isn't optional; it's the seatbelt.

Evaluation gets harder. The path is no longer fixed. Two runs of the same question can take different routes to the same answer, which breaks any test that asserts on how the system worked rather than what it produced. Your evals have to grade outcomes, not mechanisms — which, as part six argues, you should have been doing anyway.

Your tool descriptions become your routing logic. This is the subtle one. When the model chooses the tools, the sentence describing what search_contracts does is doing the job a router's if statement used to do. It deserves the same scrutiny you'd give routing code — versioned, reviewed, and tested. Most teams write those descriptions once, casually, and never look at them again, then wonder why the model keeps reaching for the wrong tool. The prose is the program.

Where it lands

Agentic retrieval is the highest-leverage change in this series, and I want to be precise about why: it's not that the model is a better retriever. It's that a fixed pipeline has to be right in advance, for every question shape, and a loop only has to be right eventually, for the question in front of it. Recoverability beats prediction.

It's also not exotic. The copilot on this site is agentic in exactly this sense — a small set of typed tools, a bounded loop, tool descriptions treated as the routing surface they are, and a hard cap on steps. No orchestrator, no sub-agents, no framework. That's usually all "agentic" needs to mean.

The frontier pushes this idea further — what if the model doesn't just choose among retrieval tools, but computes over the corpus in code? That, and the other bets on what comes after retrieval, is where the series ends. But everything up to here — the seven families, the ingestion pipeline, the model-driven loop — is the proven, in-production core. From that base, we can finally do what part one promised: choose deliberately, and prove it works.

That's part six.