When the Question Needs Every Match

This is the first module. Add it only if your users ask for lists or counts; the other modules build on the router it introduces.

A technician asks: "How many error codes does the T-7 have?"

The assistant searches, keeps the five best pieces, and the model receives five T-7 error code entries. It answers five, with a citation for each. Every citation is correct. The answer is wrong: the T-7 has six, because the sixth entry ranked below the cut.

Nothing in the answer hints at this. The model can only count what it was given, and it was given the top five because k is 5. Ask "List every display error code" and the same thing happens: five entries come back, out of ten.

This part adds a second way to answer: for questions that need every match, the system stops searching and lists.

Why a bigger k doesn't fix it

The obvious fix is to supply more results. With k set to 10, both questions happen to come out right today. But:

• The cost applies to every question. The prompt for "List every display error code" grows from about 325 tokens to 570, and every single-fact question pays for five more blocks it doesn't need. Part 7 showed what extra blocks do to precision.
• It's correct by coincidence. The eleventh error code breaks it again, and nothing will tell you. The answer looks the same whether the list is complete or not.

Search ranks. It returns the pieces most likely to be relevant, in order, and stops at a number you chose. A question like "list every…" or "how many…" asks for a set, and the only honest way to answer it is to look at every candidate and keep all the ones that match. That operation is called enumeration, and you can't do it over pieces of text that only a search can find.

A source you can enumerate: records

Some company knowledge already has a fixed shape. Every error code entry has a code, a product, and a meaning. Every row of a battery safety check table has an ID, a product, and an instruction. Search treats them as text, but they're really lists of items with fields.

So at ingestion, we also copy them into records: one record per error code entry, one per check row, each with named fields. Together they form an intermediate knowledge base, a layer between the raw documents and the questions, built for the questions the documents answer badly.

Three rules keep records trustworthy:

1. Each record points to its source section. Citations keep working: a record from section 5.3 of ERR-T7-R1 cites that section.
2. Each record keeps its document's access groups, so the access filter from Part 6 applies to records unchanged.
3. Fields come from the document's structure where possible. The parser from Part 3 already found the headings and table rows; the code and the check IDs are read from them, not guessed.

One record, as written to knowledge/records.json:

{  "id": "ERR-C3-R2-4.18-record",  "type": "error_code",  "source": "ERR-C3-R2#4.18",  "access": "staff",  "code": "E-418",  "component": "motor",  "product": "C-3",  "text": "E-418 (C-3): the motor drew too much current and assistance was switched off. Stop, switch the bike off for ten seconds, and restart. ..."}

The component field is different from the others: nothing in the document states it. Our code assigns it from the first component word in the meaning ("the motor drew too much current"). A real system would ask a model to fill such fields from the text, which is the only option when the source is prose. Either way, a filter on that field is only as good as the field, so it needs a person's review.

Checking that the records are complete

A list answer is complete relative to the records. That moves the question: are the records complete?

python -m atlas records builds them and checks them against the documents, counted a second way. It counts the error code headings and table rows in each document, and compares those counts with the records. It also scans every document for anything that looks like an error code and has no record. Here is its first run, before any curated records existed (move knowledge/curated.json aside to reproduce it):

Records: 8 error_code, 31 check (0 curated by a person)  ERR-C3-R2: 3 of 3 error code entries  ERR-T7-R1: 5 of 5 error code entries  SPEC-C3-R5: 11 of 11 check rows  SPEC-K1-R1: 8 of 8 check rows  SPEC-T7-R2: 12 of 12 check rows  NEEDS REVIEW: E-519 is mentioned in BULLETIN-2026-05 section 3 but has no entry

Here the counts match by construction, because our extraction reads the structure mechanically. With a model extracting records from prose, this comparison is what catches the items it silently dropped.

The last line is the interesting one. The firmware 3.2 bulletin says: "Firmware 3.2 adds error code E-519 on the C-3 and the T-7". The error code documents don't list it yet; they will at their next revision. Until then, any list built from those documents alone is missing a code that technicians can see on their screens. Search wouldn't have noticed either; it would just have ranked the bulletin somewhere.

No retrieval setting can fix this. The documents disagree about what exists, and a person has to decide what the records should say. Here, they add two records by hand to knowledge/curated.json, one per product, each citing the bulletin and saying why it exists:

{  "id": "CURATED-E-519-T-7",  "type": "error_code",  "code": "E-519",  "product": "T-7",  "component": "sensor",  "access": "staff",  "source": "BULLETIN-2026-05#3",  "text": "E-519 (T-7): the speed sensor signal was lost while riding. Check the magnet on the rear wheel and the sensor cable.",  "note": "Added by firmware 3.2. ERR-T7-R1 doesn't list it yet; remove once the next revision does."}

Records: 10 error_code, 31 check (2 curated by a person)

Both files live in the repository. records.json is rewritten whenever the records are rebuilt, so a change in the documents shows up as a diff in a pull request. curated.json is written by people and reviewed the same way. Knowledge the assistant relies on gets the same treatment as code: reviewed, versioned, and owned by someone.

A router: choosing the path

Records only help if list questions reach them. Something has to read the question first and decide: search for the best few, or list every match? That component is a router.

Our router takes the question and returns a proposal with three fields: the path (search or list), the record type to list, and optionally a component to filter by. Code then checks the proposal before using it, as in Part 6 with extracted fields: a record type that doesn't exist, or anything unclear, falls back to search, which is where every question went before.

def validate_route(proposed: dict) -> dict:    """Use the list path only for a record type that exists; ignore an unknown component."""    record_type = proposed.get("record_type")    if proposed.get("path") != "list" or record_type not in RECORD_TYPES:        return {"path": "search", "record_type": None, "component": None}    component = proposed.get("component")    return {"path": "list", "record_type": record_type, "component": component if component in COMPONENTS else None}

The companion code's stand-in router uses a few rules: a list word ("list", "every", "all", "how many", or "which" with a plural) plus a record type it knows. A real router asks a model to choose among the same paths and return the same three fields as structured output.

Routing adds a new way to fail: sending a question down the wrong path. "Which error code means the battery is too hot?" starts with "which", but it has one answer, and a list of every battery code would bury it. So routing gets its own test cases, like retrieval did:

  PASS                   Q17-single-stays-search  route search; all present

Every request also records a route span, so the log from Part 8 shows which path each question took and lets you review the router's choices like any other stage.

The list path, step by step

"How many error codes does the T-7 have?" now goes like this:

1. extract: product T-7, as in Part 6.
2. route: the router proposes list for error_code; validation accepts it.
   route             0.224 ms  path=list, record_type=error_code, component=None

3. records: keep every error_code record that the user may read and that fits the situation, here product T-7. No ranking, no k.
   records           0.023 ms  matches=6

4. assemble: one labeled source per record. Code, not the model, counts the matches and states the number as a known fact, and the instructions say the list is complete:
   The sources below are every record that matches the question; the known facts give their number.Answer from all of them, and don't add items that aren't listed....Known facts: 6 matching display error codes (the complete list); product T-7.
   [S1] ERR-T7-R1, section 5.1E-412 (T-7): the battery management system lost contact with one cell group. ...

5. generate and citations, as before. Each item cites its own section, and the curated E-519 record cites the bulletin.

Models miscount long lists, so the count comes from code. The model only has to write the answer around it.

If no record matches, the assistant says so: "No display error codes that you can access match this question." The wording matters. "There are none" would be a claim about records this user may not be allowed to see.

What changed

Two of the earlier questions are worth a second look.

Q04 from Part 2 asked for all eleven battery safety checks of the C-3. Search had been supplying the whole table since Part 4, so all eleven checks were present, but only because the table happened to be one block. It now takes the list path, and the check runner can say why the list is complete:

In Part 8's week of traffic, r008, "Which checks do I need before delivering a T-7?", was the false alarm: a weak search that happened to supply the right table. It's a list question, and now the router sends it to the list path, which returns all twelve T-7 checks. Replayed, the week has no weak searches left:

Routes: search 23, list 1
Weak searches (best score < 0.5, no exact match): 0

The held-out set got one list question, "Which battery safety checks apply to the K-1?", which the list path answers with all eight K-1 checks. Development recall stays at 1.00.

The system now

flowchart LR
  Q[Question] --> E[Extract product and date]
  E --> R{Router}:::new
  R -- "search" --> S[Exact + keyword + vector, fuse, rerank, top k]
  R -- "list" --> L["Every matching record (access + situation filters)"]:::new
  D[(Documents)] --> X[Extract records]:::new
  X --> K[(Records + curated records)]:::new
  K --> L
  X --> C{{"records: completeness report"}}:::new
  S --> P[Prompt] --> M[Model]
  L --> P
  classDef new stroke-width:3px

Stop here if

Your list and count questions are about a few kinds of things with a fixed shape (error codes, checks, products, contacts), and someone can own their records. Each new record type is a small, deliberate addition: a definition, an extraction, a completeness check, and test cases for routing to it.

This module doesn't cover a list question about something without records. "Which bikes are affected by the charger recall?" still goes to search, and that's fine as long as one passage answers it. Records also give you the items, not arithmetic over them. A question like "what was the pass rate across all charge-cycle tests?" needs numbers computed, not listed. That's Part 10.