Choosing the Evidence That Applies

Two customers call the shop on the same day. One bought a C-3 in March 2026, the other in March 2025. Both ask about their battery warranty. After Part 5, here is what search returns for each:

"...a C-3 bought on 15 March 2026?"        "...a C-3 bought on 15 March 2025?"0.582  WARR-R2-2-p0   (2026 policy)        0.582  WARR-R2-2-p0   (2026 policy)0.582  WARR-R1-2-p0   (2025 policy)        0.582  WARR-R1-2-p0   (2025 policy)0.429  WARR-S2-R1-2-p0 (scooter policy)    0.429  WARR-S2-R1-2-p0 (scooter policy)

The results are identical. The two C-3 policies tie, and the scooter's policy comes third. The one fact that decides the answer, the purchase date, changed nothing.

Search is working as designed: it measures how well a passage matches the question, and both policies match a question about C-3 battery warranties equally well. What we need is the passage that applies to this customer's bike, and that depends on facts search doesn't look at.

The facts are already in two places

The question carries the facts that matter: the product (C-3) and the purchase date (15 March 2026).

The documents carry the matching facts as metadata. Part 1 showed this chunk, stored with fields that came from the company's document system, not from the text:

meta={"title": "C-3 battery warranty, revision 2", "product": "C-3",      "access": "employees", "applies_from": "2026-01-01"}

The 2025 policy has applies_from: 2025-01-01 and applies_through: 2025-12-31. The scooter policy has product: S-2. These fields have been in the index since Part 1, unused. This article connects them to the question.

Step 1: pull the facts out of the question

A language model reads the question and fills in a small set of fields. A typical instruction: "Copy the product and purchase date from the question. Return null for anything not stated. Don't guess." For our question it returns:

{"product": "C-3", "purchase_date": "2026-03-15"}

The companion code uses a stand-in that finds these with patterns, but the contract is the same: the extractor proposes values.

Step 2: check them in code

A proposed value is not yet a fact. The application checks each field before using it:

def situation_from(fields, known_products):    product = fields.get("product")    if product not in known_products:        product = None    on_date = None    if fields.get("purchase_date"):        try:            on_date = date.fromisoformat(fields["purchase_date"])        except ValueError:            on_date = None    return Situation(product=product, on_date=on_date)

An unknown product or an unreadable date counts as missing, never as a guess. For "What does error E-417 mean?", the stand-in extractor proposes E-417 as the product, because it looks like a model name. Atlas Bikes sells the C-3 and the S-2, so validation drops it, and the question is searched without a product filter. Without this check, the filter would have removed every document about the C-3 and found nothing. The model proposes what the question says; code decides which proposals count as facts.

Step 3: search only what applies

The validated facts become a filter that every search respects:

def allows(self, chunk):    product = chunk.meta.get("product")    if self.product and product and product != self.product:        return False    start, end = chunk.meta.get("applies_from"), chunk.meta.get("applies_through")    if self.on_date and start and self.on_date < date.fromisoformat(start):        return False    if self.on_date and end and self.on_date > date.fromisoformat(end):        return False    return True

A document without these fields is always allowed: the filter only excludes documents that say they apply elsewhere.

The filter runs inside each search, before ranking, rather than on the results afterward. If it ran afterward, inapplicable policies would still take up the five slots, and we could end up with fewer than five useful passages. This is usually called metadata filtering, and most vector databases support it directly.

Now the two customers get different evidence:

"...a C-3 bought on 15 March 2026?"        "...a C-3 bought on 15 March 2025?"0.577  WARR-R2-2-p0   (2026 policy)        0.577  WARR-R1-2-p0   (2025 policy)

The other policies are gone, not ranked lower: they never become candidates. Nothing here is specific to warranties. The same filter works for any document that applies to a product, a region, a contract, or a date range, as long as the metadata exists.

Step 4: ask when the facts are missing

"How long is the battery warranty?" names no product and no date. The filter from Step 3 has nothing to filter by, so search returns all three policies, tied. Any single answer would be a guess. A person at the desk would ask which bike it is and when it was bought. The assistant should do the same.

This check runs on every question, right after search and before the model is called. It uses two things the pipeline already has at that point:

• The situation: what we know from the question, meaning the validated product and purchase date from Step 2. Either can be empty.
• The hits: the pieces search returned, best first.

The rule, in words: look at the best hit. If its document doesn't depend on a date, nothing is missing, and the assistant answers as usual. If it does depend on a date (our warranty policies have an applies_from field; the error catalog doesn't):

1. If we don't know the purchase date, ask for it.
2. If the hits include documents for more than one product and we don't know the product, ask for that too.

The same rule in code:

def missing_details(situation, hits):    # The best hit doesn't depend on a date: nothing to ask.    if not hits or not is_date_scoped(hits[0]):        return []    missing = []    # Documents for several products matched, and the question didn't say which.    products = {c.meta.get("product") for c in hits if is_date_scoped(c)}    if situation.product is None and len(products) > 1:        missing.append("product")    # The document depends on a date we don't have.    if situation.on_date is None:        missing.append("purchase date")    return missing

And where the pipeline calls it, between search and the model:

missing = missing_details(situation, [chunk for _, chunk in hits])if missing:    trace.status = "clarify"    trace.answer = f"Which {' and '.join(missing)} is this about?"    return trace

If anything is missing, the assistant replies with a question and stops there; the model is never called:

How long is the battery warranty?  -> Which product and purchase date is this about?
What battery warranty applies to a bike bought on 15 March 2026?  -> Which product is this about?

The second question already has a date, so only the product is missing: on that date, both the C-3 and the S-2 policies apply.

Nothing in the rule mentions warranties. It reacts to metadata: any document with a validity period triggers it, and documents without one pass straight through. For "What does error E-417 mean?", the best hit is the error catalog entry, which has no date, so the assistant answers without asking anything.

Step 5: citations that code can check

An answer should say where it came from, so the reader can open the document and check. In Part 1, the prompt asked the model to "identify the source", so the model typed source names itself. A model can get a name slightly wrong, or invent one, and nothing would catch it.

Now citing happens in three steps, and the model only does the middle one.

1. The application labels the sources. While building the prompt, it gives each block a short label and remembers what each label stands for:

S1 -> WARR-R2, section 2S2 -> SPEC-C3-R5, section 6

2. The model cites labels. The prompt shows the labelled blocks and asks for the label beside each claim:

Cite the source label, for example [S1], beside each supported claim....Known facts: product C-3; purchase date 2026-03-15.
[S1] WARR-R2, section 2C-3 batteries purchased on or after 1 January 2026 carry a warranty of 3 years ...

The model's answer then contains labels, not names. Our stand-in model returns:

C-3 batteries purchased on or after 1 January 2026 carry a warranty of 3 yearsfrom the purchase date. [S1]

3. The application turns labels into links. It looks up each label it remembered in step 1 and replaces it:

C-3 batteries purchased on or after 1 January 2026 carry a warranty of 3 yearsfrom the purchase date. [WARR-R2, section 2](/documents/WARR-R2#section-2)

The model never writes a document name or a link, so it can't invent one. If it cites a label that wasn't in the prompt, say [S7], the lookup fails and the application withholds the answer instead of showing a broken source:

The answer cited unknown sources ['S7'] and was withheld.

One limit remains. A valid label proves the source exists and was given to the model. It doesn't prove the source supports the sentence next to it: the model could attach [S1] to a claim that S1 never makes. Checking that is part of measuring answers, in Part 7.

Something still visible

Look again at what the 2026 question receives. After the policy comes a second block: the battery safety checks table. The system keeps five pieces, the policy filled one slot, and the next-closest matches were rows of that table. Nearest isn't relevant, and the model has to ignore it. Part 7 measures how much of what reaches the model is noise; a minimum relevance score is one way to reduce it.

The system now

flowchart LR
  Q[Question] --> EX{{"Extract product + date"}}:::new --> V{{"Validate in code"}}:::new
  V --> F{{"Filter every search"}}:::new
  F --> S["Search: exact + keyword + vector, fusion, rerank"]
  S --> C{"Details missing?"}:::new
  C -- yes --> ASK[Ask the user]
  C -- no --> B[Blocks labeled S1, S2...] --> M[Model] --> L{{"Labels to links"}}:::new --> A[Answer]
  classDef new stroke-width:3px

Questions without a date, like the error code, PIN, slide, and figure questions, flow exactly as before: the filter only removes other products' documents, and nothing asks for clarification.

This is the complete core

Six articles in, the assistant can answer questions that have one answer, found in a passage, a table, a figure, or a slide; that use the documents' words or the user's own; that include exact codes; and whose right answer depends on a product or a date. It asks when it can't tell, and its sources can be checked.

Many real assistants need nothing more. Two cases still fail, and they are a different kind of problem:

Listing every safety check and calculating a pass rate are operations search can't perform, however good it gets. Parts 9 to 13 add those capabilities as independent modules. Before adding anything, though, we need a better way to know how well the core works.

Stop here if

Stop here if your users ask questions with one answer each, and the right document can depend on the product, date, or other stated condition. This is a complete system.

Part 7 builds on the check runner from Part 2 to measure the core properly: whether the right evidence arrives, how much noise comes with it, whether answers are correct, and how stable they are.