When a Useful Answer Isn't a Correct Answer

Part 1 ended with the delivery team asking for every battery safety check on the C-3. The assistant received eight of the eleven. Each passage was accurate and had a source. The list looked complete, and a bike could have left the shop with its charging port open.

We only noticed because we knew there were eleven checks. That raises an uncomfortable question: how many other questions fail without anyone noticing?

Before changing anything in the system, we need to find out. This article collects real questions, writes down what a correct answer looks like for each, and turns that into a small program that checks the assistant automatically.

Ask the people who will use it

The best test questions come from the people who will ask them. At Atlas Bikes that means the service desk, the workshop, the delivery team, and dealers. Collect questions from support tickets, chat messages, and by sitting next to someone for an hour. Keep the original wording, including the vague parts: real users don't phrase questions the way documentation is written.

For each question, ask the person how they answer it today. Which document do they open? How do they know the answer is complete? What would a wrong answer cost them?

Here are five questions from that exercise, run through the Part 1 assistant.

"How long is the battery warranty?" The model receives three policies: the S-2 scooter's (ranked first), the C-3 policy for 2025 purchases, and the C-3 policy for 2026 purchases. The question names neither a product nor a purchase date. Any single answer would be a guess. A colleague at the desk would ask: which bike, and when did you buy it?

"What does error E-417 mean, and what should I check?" These are the five passages the model receives:

0.231  SPEC-C3-R5-6-chunk-3        battery safety checks R01, R020.118  ASSEMBLY-C3-R3-4.3-chunk-3  "Check that turning the handlebar..."0.109  SPEC-C3-R5-6-chunk-4        battery safety checks R03 to R050.107  ASSEMBLY-C3-R3-4.2-chunk-1  "...check that the mudguard does not touch the tyre"0.098  SERVICE-C3-R4-7-chunk-1     "...an authorized dealer must check..."

Every one contains the word "check". The error code entry, the only passage that answers the question, isn't there. Search matched the least important word in the question and ignored the most important one.

"What does it mean when the C-3 display blinks twice?" The answer is in the speaker notes of a training slide. The model receives the slide, but without its notes.

"What share of bikes passed final inspection without rework?" The answer isn't written in any document. It has to be calculated from inspection records.

"List all battery safety checks for the C-3 under revision 5." Eight of eleven, as in Part 1.

Five failures, five different causes

All five would be reported the same way: "the assistant got it wrong". That label is useless for fixing anything, because each failure happened in a different place:

flowchart LR
  Q[Question] --> S[Search]
  D[Documents] --> PR[Preparation] --> I[(Index)] --> S --> P[Prompt] --> M[Model]
  F1["Blinks twice: notes dropped"] -.-> PR
  F2["E-417: entry not found"] -.-> S
  F3["Warranty: no purchase date"] -.-> Q
  F4["All checks: closest 5, not all 11"] -.-> S

Notice what isn't on the list: the language model. In every case, the model never received what it needed. Swapping in a smarter model would fix none of these. That is common in practice and worth checking first: before blaming the model, look at what it was given.

Write down what correct means

For each question, write down the behavior you expect. Keep it short and specific to the question:

We'll call this an answer contract: what a correct response must do. For the warranty question, the correct response is a clarifying question.

Then turn each contract into something a program can check. In the companion code, every case lives in cases/cases.json. Here is the safety checks question:

{  "id": "Q04-every-check",  "question": "List all battery safety checks for the C-3 under revision 5.",  "expected_from": "part-09",  "behavior": "All eleven checks R01-R11, with a guarantee that the list is complete rather than a lucky retrieval.",  "checks": [    {"type": "includes_ids", "ids": ["R01", "R02", "R03", "R04", "R05", "R06", "R07", "R08", "R09", "R10", "R11"]},    {"type": "complete_by", "route": "enumeration"}  ]}

The checks look at the evidence: which passages, IDs, or facts reached the model. They don't grade the model's prose. That choice is deliberate. Evidence checks give the same result every time, run in milliseconds, and tell you where a failure happened. Judging whether the wording of an answer is right needs a person or another model, which Part 7 covers.

The expected_from field records which planned change should fix the case. In this repository that is the article number. In a real project it could be a ticket or a milestone. It turns the case file into a roadmap.

A runner that labels each case

The runner asks every question, applies its checks, and compares the result with expected_from. The checks themselves are a few lines each:

def evaluate(trace, check):    refs = [chunk.ref for chunk in trace.context]    supplied = "\n".join(chunk.text for chunk in trace.context)    kind = check["type"]    if kind == "includes_refs":        missing = [r for r in check["refs"] if r not in refs]        return not missing, f"missing {missing}" if missing else "all present"    if kind == "includes_text":        ok = check["text"] in supplied        return ok, f"'{check['text']}' {'supplied' if ok else 'not supplied'}"    ...

The label depends on whether the case should already pass:

if passed:    label = "PASS" if current >= due else "EARLY PASS"else:    label = "REGRESSION" if current >= due else f"KNOWN GAP -> {case['expected_from']}"

A known gap is a failure we already understand and plan to fix. A regression is a case that should pass and doesn't: something we changed broke it. That distinction is what makes the runner useful day to day. Known gaps are a to-do list; regressions stop a release.

Here is the full output for the current system:

Two of fifteen cases pass. That looks bad, and it is the most useful result so far: it shows exactly what to work on, and in which order. The rest of the series works through this list.

Keep a trace of every answer

When a case fails, you need to see what happened. Running the checks with --save-traces writes one file per question with everything the pipeline did: the question, the passages retrieved with their scores, the prompt, the answer, and its status. Here is the start of the trace for E-417:

{  "question": "What does error E-417 mean, and what should I check?",  "retrieved": [    ["SPEC-C3-R5-6-chunk-3", 0.231],    ["ASSEMBLY-C3-R3-4.3-chunk-3", 0.118],    ...  ],  "status": "answered"}

The error entry isn't among the retrieved passages, so the investigation starts at search, not at the prompt or the model. Without the trace, you would be guessing.

The system now

flowchart LR
  C[(cases.json)] --> R{{Check runner}}:::new
  R -- each question --> A[Assistant from Part 1]
  A -- trace --> R
  R --> O[PASS / KNOWN GAP / REGRESSION]
  A -. "--save-traces" .-> T[(traces/)]
  classDef new stroke-width:3px

The assistant itself hasn't changed. What changed is that every later change can now be measured against the same questions, and anything it breaks shows up as a regression.

Not a stopping point

Every system needs this, so this isn't a place to stop. Keep the case file and the runner whatever you build next.

The first three known gaps share a cause: the information was damaged or dropped before it reached the index.

Part 3 starts there, with what happens between a raw file and the index.