When Search Misses the Evidence

A mechanic asks:

What does error E-417 mean, and what should I check?

After Part 4, these are the five pieces search returns:

0.189  SPEC-C3-R5-6-table0-R03     battery safety check R030.188  TRAIN-C3-2-search           training slide: display blink patterns0.177  ERR-C3-R2-4.18-p0           error E-4180.177  SPEC-C3-R5-6-table0-R02     battery safety check R020.174  ERR-C3-R2-4.71-p0           error E-471

Two error code entries made it, E-418 and E-471. The one the mechanic asked about, E-417, did not. E-417 means the battery is too hot to charge. E-471 means it is too cold. A model handed E-471 could confidently tell the mechanic to warm up a battery that is already overheating.

This is a different failure from the ones before. The E-417 entry is in the index, intact, in its own section. Search simply didn't pick it.

Why vector search can't see the code

Vector search compares meaning. To an embedding model, "E-417" carries almost no meaning: it is a code, not a word. Our stand-in embedding model ignores anything containing digits, which exaggerates the effect, but real embedding models show the same weakness on identifiers, part numbers, and codes that differ by one character. E-417, E-418, and E-471 all look like "some error code about the battery".

So the question is matched on its other words: "error", "mean", "check". The battery safety checks table matches "check" well. The E-417 entry, which never says "check", ranks sixth, one place outside the five passages we keep.

Keyword search sees exactly what vectors miss

Older than embeddings, and still everywhere, is keyword search: score each passage by the words it shares with the question. The standard scoring method is BM25. It gives more weight to rare words than common ones, rewards repetition with diminishing returns, and avoids favoring long passages just for being long. "E-417" appears in exactly one passage, which makes it the rarest term in the question, and the one that counts most.

The heart of BM25 is one line per matching term (keyword.py):

idf = math.log(1 + (n - doc_freq[term] + 0.5) / (doc_freq[term] + 0.5))score += idf * tf * (self.k1 + 1) / (tf + self.k1 * (1 - self.b + self.b * length / avg_len))

idf is the rarity of the term across all passages, tf is how often it appears in this one, and k1 and b tune the diminishing returns and the length adjustment. Keyword search puts the E-417 entry first.

Keyword search has the opposite weakness. Ask "How do I get back into the display after forgetting the PIN?" and it ranks the section that says "PIN" four times first, "Changing the display PIN", above the recovery procedure the dealer needs. Vector search gets this one right: it matches "forgetting" to "unavailable" and puts the recovery section first.

Each method fails where the other succeeds. So use both.

Hybrid search: combine the two lists

Hybrid search runs keyword and vector search side by side and merges their results. The two methods produce scores on different scales (BM25 scores here range up to about 5, vector similarities stay below 1), so adding them directly would let one dominate. Reciprocal rank fusion (RRF) ignores the scores and uses only positions: each list gives a passage 1 / (60 + rank), and the totals decide the merged order.

def reciprocal_rank_fusion(*rankings, c=60):    scores, by_id = {}, {}    for ranking in rankings:        for rank, chunk in enumerate(ranking, start=1):            scores[chunk.id] = scores.get(chunk.id, 0.0) + 1 / (c + rank)            by_id[chunk.id] = chunk    return [by_id[i] for i in sorted(scores, key=scores.get, reverse=True)]

For the E-417 question, the fused order starts with check R03, then the training slide, then the E-417 entry in third place. Third is inside our five, so the right evidence now reaches the model. It still isn't first.

When the question names an exact code, look it up

For some questions, search isn't the right operation at all. When a mechanic types a code that exists in the catalog, there is exactly one right entry. Part 3 kept each error code as a field from the HTML page (error_code), so the application can look it up directly, the same way a database looks up a row by its key:

codes = set(re.findall(r"\b[A-Z]+-[A-Z]?\d+\b", question))exact = self.index.lookup("error_code", codes, groups)

Exact matches go first. Everything else is ranked as before. The plain-text extraction from Part 1 had thrown that HTML attribute away; this is the first time the conversion work from Part 3 pays off in retrieval.

A second, closer look: reranking

Search methods score each passage on its own, using precomputed vectors or word counts. A reranker takes the shortlist and reads the question and each passage together, the way a person would skim results. The usual implementation is a cross-encoder, a model that scores one question-passage pair at a time. That is too slow to run over a whole index, but fine for a shortlist of twenty.

The whole retrieval step now reads (pipeline.py):

codes = set(re.findall(r"\b[A-Z]+-[A-Z]?\d+\b", question))exact = self.index.lookup("error_code", codes, groups)
lexical = [c for _, c in self.keywords.search(question, groups, self.candidates)]dense = [c for _, c in self.index.search(self.embedder.embed(question), groups, self.candidates)]shortlist = [c for c in reciprocal_rank_fusion(lexical, dense) if c not in exact]
scores = self.reranker.score(question, [c.search_text or c.text for c in shortlist])ranked = sorted(zip(scores, shortlist), key=lambda pair: pair[0], reverse=True)return ([(float("inf"), c) for c in exact] + ranked)[: self.k]

There are now two limits: how many candidates each search contributes (10), and how many pieces we keep (5). Raising the first gives the reranker more to choose from; raising the second gives the model more to read.

flowchart LR
  Q[Question] --> X{{"Exact code lookup"}}:::new
  Q --> K{{"Keyword search (BM25)"}}:::new
  Q --> V[Vector search]
  K --> F{{"Fusion (RRF)"}}:::new
  V --> F
  F --> R{{Reranker}}:::new
  X --> T[Top 5 pieces]
  R --> T
  T --> B[Blocks] --> P[Prompt] --> M[Model]
  classDef new stroke-width:3px

The E-417 question now receives the right entry first:

    inf  ERR-C3-R2-4.17-p0  0.400  TRAIN-C3-2-search  0.200  SPEC-C3-R5-6-table0-R03

The PIN question still finds the recovery procedure, now second behind the PIN-changing section, which points to it.

Non-text content needs nothing special here. "Which check covers the battery vent holes?" matches the row for check R08 first, and Part 4's step hands the model the whole table.

A reranker can make things worse

The reranker in the companion code is a stand-in, like the embedding model, so we had to decide how it scores passages. Our first attempt made a mistake that real rerankers can make too, and the check runner caught it.

Take the question from Part 3:

What does it mean when the C-3 display blinks twice?

Before reranking, things look good. Both searches rank the right training slide first, the one whose notes explain two blinks.

Our first reranker gave each passage two kinds of points:

• 3 points for each code it shares with the question. The idea was that codes like E-417 are precise, so a match should count a lot.
• A small score for shared words, such as "blink" or "display", lower for long passages.

The question contains one code: "C-3", the bike's model name. Here is how the top candidates scored:

The only passage about blinking scored lowest. Six passages that merely mention the model name beat it, because the slide itself never says "C-3", while half of all searchable pieces do. The slide dropped to seventh place, out of the five the model receives, and the check runner reported:

The mistake was treating every code as precise. "E-417" appears in one passage, so matching it means a lot. "C-3" appears almost everywhere, so matching it means almost nothing. A second attempt gave smaller bonuses to common codes, but still added them on top of the word score, and failed the same way.

The version that shipped counts a code as one more word of the question, worth little when it is common. The blink slide now scores 0.50 and the six model-name passages 0.22 each, so the slide is back in first place.

Real rerankers are trained models, and they too can misjudge which words carry the meaning in your documents. Without the case file from Part 2, the broken reranker would have shipped, and answers about blink patterns would have quietly gotten worse.

The system now

Every earlier case still passes.

Stop here if

Stop here if your users ask single-answer questions, some with exact codes or names and some in their own words, and your documents don't come in versions that apply to different situations.

Atlas Bikes' warranty documents do. Ask about the battery warranty for a C-3 bought in March 2026, and the model still receives three policies:

The 2025 policy and the scooter's policy are good matches for the question. They are also wrong for this bike. Part 6 is about choosing, among the evidence that matches, the evidence that applies.