Four Ways to Retrieve Text, and How Each One Breaks

Part one argued that RAG decomposes into five decisions, and that a vector database is one setting of one of them: selection. This part is about that dial, and about the four families most teams actually choose between.

What follows is not a ranking. These are instruments with different characteristic failures, and choosing well means knowing which failure your questions can survive.

So I'll be specific about how each one breaks. The strengths are what vendors publish. The failures are what you inherit.

1. Inline everything

The simplest retrieval strategy is not retrieving.

If the corpus fits in the context window, retrieval is the identity function. Send all of it, every time. No index, no chunk size, no similarity threshold, no sync job.

Good at: nothing gets missed. Recall is total by construction. The model sees the whole corpus at once, so cross-document reasoning works without any special handling. And there's no drift — no embedding model to upgrade, no index to rebuild when content changes, nothing to babysit.

Breaks when: you pay for the entire corpus on every request, so cost scales with corpus size times traffic. Latency grows with input length. And quality degrades well before the hard limit. Models attend unevenly across long inputs — they recall less reliably from the middle than the edges, the effect Liu et al. named "lost in the middle". A corpus that technically fits can still answer worse than a smaller, well-chosen slice of it.

One thing has changed this arithmetic more than people have absorbed: prompt caching. If the corpus is a stable prefix, you pay full price once and a fraction afterwards. The practical ceiling sits considerably higher than it did two years ago.

Reach for it when: the corpus is small and stable, or the unit of work is one large document — a contract, a spec, an incident file. The copilot on this site works this way. About two thousand tokens of entity facts, inlined on every request, no retrieval step at all for most questions.

2. Lexical search

BM25 and its relatives, running inside Postgres full-text, OpenSearch, or SQLite FTS. Decades of tuning, all of it CPU-cheap.

It gets underestimated because "keyword search" sounds like matching one word at a time.

It isn't. BM25 scores each document against all of the query's terms together, and three behaviours do most of the work:

• Rare terms dominate. A term's weight is inverse to how many documents contain it. In a payments knowledge base, "payment" appears everywhere and contributes almost nothing; "chargeback" appears in a handful of documents and effectively decides the ranking. Nobody configures that. It's derived from the corpus itself.
• Repetition saturates. A page that says "timeout" forty times is not twenty times more relevant than one that says it twice. The curve flattens on purpose — that's what stops keyword-stuffed pages from winning.
• Length is normalized. The same match counts for more in a short, focused document. A 200-word runbook can outrank an 80-page handbook that mentions the term in passing.

Here's what that looks like on the query payment gateway timeout after retry. The numbers are invented for the example — the mechanics are what's real:

The winning document isn't the one containing the most query words. It's the short one where timeout and retry occur together. Two of the four terms did nearly all the ranking work, and the engine figured that out from the corpus without being told anything about payments.

Above that sits an analyzer layer most people never look at: stemming, so "retries" matches "retry"; phrase and proximity scoring, so "payment gateway" can rank as a unit; boolean must / should / must-not; per-field boosting so a title match beats a body match. A well-tuned lexical index is a great deal more than string matching.

Good at: exact tokens. Error codes, SKUs, ticket IDs, function names, version strings, proper nouns. It's fast, needs no GPU, and it's explainable — you can point at the matched terms and say exactly why a document ranked where it did. And there's no model to deprecate, so nothing ever forces a reindex.

Breaks when: the asker's words differ from the document's words. All that machinery is sophisticated about terms and completely blind to meaning. "How do I reset my password" against a page titled "Credential recovery procedure" shares no terms and scores near zero. No amount of BM25 tuning fixes that, because the failure isn't in the ranking. It's in the vocabulary. Synonym lists patch it, then rot.

Reach for it when: the corpus is identifier-heavy — code, logs, tickets, legal citations, part numbers. And more often than teams expect, reach for it as a baseline. If an embedding pipeline can't beat BM25 on your golden set, the pipeline has a bug, not a tuning opportunity. Skipping that comparison is one of the most common unforced errors in this space.

3. Dense vectors

The one everyone means. Embed chunks and the query into a shared space; return the nearest neighbours.

Good at: exactly what lexical can't do — matching meaning across different vocabulary. Paraphrase, synonymy, and with the right model, across languages. Approximate nearest-neighbour indexes keep search fast over millions of chunks. When a corpus is large, unstructured, heterogeneous, and users phrase questions unpredictably, nothing else takes you this far this cheaply.

That situation is real and extremely common, and dense retrieval earns its reputation there. The argument in this series is against defaulting to it. Not against using it.

Breaks when:

• Exact tokens matter. Rare identifiers get blurred toward their neighbourhoods. A search for error code X-4021 can happily return X-4027.
• The query contains logic. "Contracts without an arbitration clause," "incidents after March." Embeddings encode topic, not negation or constraint.
• Completeness is required. Top-k is a fixed budget. Forty documents match, you retrieve k, and the model reports on those as though they were the population. This is the failure that broke the spec assistant in part one.
• Nothing is relevant. Similarity always returns its k nearest, however distant. There's no natural "I found nothing" signal. Thresholds help; the right threshold is corpus-specific, and it drifts.
• You needed certainty. ANN indexes trade recall for speed by design. The retriever misses things and never tells you.

Two operational costs get underestimated too. Chunking decisions dominate quality more than the choice of database. And an embedding upgrade means reindexing everything — the embedding model is a long-term commitment, not a config value.

4. Hybrid, then rerank

Lexical and dense fail on different queries. Their errors are largely uncorrelated. That's the whole reason running both and merging recovers most of what either misses alone.

Fusion doesn't need to be clever. Reciprocal Rank Fusion scores purely by rank position:

from collections import defaultdict
 
# Reciprocal Rank Fusion — no tuning, no score normalization.
def rrf(rankings, k=60):
    scores = defaultdict(float)
    for ranking in rankings:            # e.g. [bm25_hits, dense_hits]
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] += 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

BM25 scores and cosine similarities aren't on comparable scales. RRF sidesteps that without anyone having to invent a weighting.

Then rerank. A cross-encoder scores the query and a candidate together, instead of embedding them independently. Substantially more accurate, far too slow to run across a whole corpus. So the shape is: retrieve broadly with both methods and a generous k, then rerank narrowly. Fifty candidates down to five.

The common practitioner finding — and the first thing I'd test — is that adding a reranker buys more than upgrading to a better embedding model. Treat that as a hypothesis to check against your own golden set, not a law.

Breaks when: you forget it's two indexes to keep in sync, plus a model hop of latency on every query. It's the right default for general-purpose text search. It is not free.

What these four have in common

They're the families most teams choose between, and they share an assumption so basic it usually goes unstated: retrieval means finding the text most similar to the question. Inlining skips the finding. The other three disagree about how to measure similarity. But all four treat the corpus as a pile of text to be matched against a query.

For a great many questions, that assumption holds.

For others, it fails completely. The counts. The filters. The exhaustive listings. The multi-hop connections. The "what is this corpus about" questions. No amount of similarity reaches those, because the answer isn't sitting in any single passage waiting to be matched.

Three more families abandon the assumption entirely. Part three is about those.