When the Answer Must Be Calculated

Add this module only if your users ask for numbers that have to be computed.

The quality manager asks: "What share of bikes passed final inspection without rework across batches A and B?"

The search path from Part 6 handles this well. Run with Part 9's code on the inspection workbook, which this part's checkpoint adds to the collection, it finds the definition of the pass rate and the inspection table, and supplies both:

[S1] QA-INSPECT-2026, section 11 DefinitionsPass rate: the share of inspected bikes that passed final inspection without rework. ...
[S2] QA-INSPECT-2026, section 22 Final inspection by batch| Batch | Product | Month | Bikes inspected | Passed without rework | Pass rate || --- | --- | --- | --- | --- | --- || A | C-3 | 2026-07 | 100 | 80 | 80% || B | C-3 | 2026-08 | 10 | 9 | 90% || C | T-7 | 2026-07 | 40 | 36 | 90% |...

Retrieval did its job: everything needed to answer is in the prompt. What's left is arithmetic, and the table makes the wrong arithmetic easy. Batch A passed 80%, batch B passed 90%, and the obvious answer is 85%.

It's wrong. Batch A had 100 bikes and batch B had 10. Averaging the two rates gives the small batch as much weight as the large one. The question asks about bikes, so add the bikes first:

passed without rework   80 + 9   =  89inspected              100 + 10  = 110pass rate               89 / 110 = 80.9%

A language model might get this right. It might also average, round differently, pick up batch C by mistake, or get it right today and wrong next week. For a number someone will put in a report, "might" isn't good enough. This part makes sure the arithmetic is never the model's job.

The metric, written down once

Every rate has two counts: the numerator (what counts as a success, here bikes that passed without rework) and the denominator (what you count out of, here bikes inspected). The definition in the workbook says it in words. The code says it as a formula:

METRICS = {    "pass_rate": Metric(        label="pass rate", record_type="inspection", numerator="passed", denominator="inspected",        counts="bikes passed final inspection without rework", definition="QA-INSPECT-2026#1",    ),}

That's the whole definition: which records to read, which field to add up for the numerator, which for the denominator, and which document section defines it. Every request for this metric uses the same formula, which a person can read, test, and agree with the team that owns the number.

Numbers as records

Part 9 copied error codes and checks into records with fields. Inspection rows become records the same way, one per batch, with the counts as fields:

Batch A (C-3, 2026-07): 100 bikes inspected, 80 passed without rework.

One column is deliberately left out: the table's own "Pass rate". It's derived from the counts, and storing it would invite exactly the mistake we're avoiding. Rates are computed, never read.

Rebuilding the records gives the completeness report from Part 9, and it has something to say:

Records: 10 error_code, 6 inspection, 31 check (2 curated by a person)  QA-INSPECT-2026: 6 of 6 inspection rows  NEEDS REVIEW: batch F in QA-INSPECT-2026 has no passed count

Batch F is still being inspected, and its passed count is blank.

A third route

The router from Part 9 gets a third path, calculate. When a question asks for a share, a percentage, or a rate, it proposes the metric and any batches the question names:

route             path=calculate, metric=pass_rate, batches=['A', 'B']

Validation works as before: a metric that isn't defined falls back to search. So does a question about the metric rather than its value. "How is the pass rate defined?" mentions the pass rate, but it wants the definition, not a number, and it has its own test case:

  PASS                   Q20-definition-stays-search route search; all present

The calculate path, step by step

calculate: code selects the inspection records in scope, with the same access and situation filters as everything else, adds the counts, and divides.
```
calculate         status=ok, metric=pass_rate, scope=['A', 'B'], numerator=89, denominator=110, value=80.9
```

assemble: the model receives the result as a sentence, the definition, and the records used, each with a label for citation. The instructions tell it to report the result as given and not recompute it:

Code computed the result below from the listed records.Report the result exactly as given and explain what was counted. Don't recompute it....Result (computed by code): 89 of 110 bikes passed final inspection without rework: a pass rate of 80.9% (batches A, B).
[S1] QA-INSPECT-2026, section 11 DefinitionsPass rate: the share of inspected bikes that passed final inspection without rework. ...
[S2] QA-INSPECT-2026, section 2Batch A (C-3, 2026-07): 100 bikes inspected, 80 passed without rework.
[S3] QA-INSPECT-2026, section 2Batch B (C-3, 2026-08): 10 bikes inspected, 9 passed without rework.

generate: the model writes the answer around the result. It gets the counts and the scope along with the percentage, so it can explain what was counted, and a reader can check it.
numbers: a check described below.
citations, as before.

The answer with our stand-in model, which repeats the result and quotes each source:

- 89 of 110 bikes passed final inspection without rework: a pass rate of 80.9% (batches A, B).- Pass rate: the share of inspected bikes that passed final inspection without rework. [QA-INSPECT-2026, section 1]- Batch A (C-3, 2026-07): 100 bikes inspected, 80 passed without rework. [QA-INSPECT-2026, section 2]- Batch B (C-3, 2026-08): 10 bikes inspected, 9 passed without rework. [QA-INSPECT-2026, section 2]

Scope can also come from the situation. "What share of T-7 bikes passed final inspection without rework?" names no batches, but the extractor from Part 6 finds the product, and the calculation keeps the T-7 batches C and D: 56 of 65, 86.2%. Averaging their rates (90% and 80%) would again have given 85%.

Checking the model's numbers

The instructions say not to recompute. A real model usually complies, but "usually" is the problem again. So after generation, code checks the answer: every number in it must appear somewhere in the prompt, in the result or in the sources. A number that appears nowhere was invented, misremembered, or computed by the model.

def unsupported_numbers(answer: str, prompt: str) -> list[str]:    """Numbers in the answer that appear nowhere in the result or the sources, such as an averaged rate."""    answer = re.sub(r"\[S\d+\]", "", answer)    allowed = set(re.findall(r"\d+(?:\.\d+)?", prompt))    return sorted(set(re.findall(r"\d+(?:\.\d+)?", answer)) - allowed)

To see it work, the tests use a stand-in model that ignores the instructions and averages: "The pass rate across batches A and B is 85% [S2][S3]." The trace:

calculate   status=ok, numerator=89, denominator=110, value=80.9generate    model=AveragingGeneratornumbers     unsupported=['85']

The answer is withheld: "The answer contained numbers the calculation doesn't support ['85'] and was withheld." In production, that request also lands in the review queue from Part 8.

The check is deliberately simple. It catches invented numbers, not misused ones: an answer that calls 89 the number of bikes inspected would pass. It's a cheap guard for the most common failure, not a proof of correctness.

When there's no result

"What share of K-1 bikes passed final inspection without rework?" covers batches E and F. Batch F's passed count is blank.

There are two tempting shortcuts. Leaving F out gives 27 of 30, 90%, which answers a question about batch E. Counting the blank as zero gives 27 of 42, 64.3%, which says twelve bikes failed that may well have passed. Both produce a confident number for a different question. So a blank count means no result:

calculate   status=incomplete_data, metric=pass_rate, incomplete=['F']

The pass rate can't be computed: batch F has no count recorded yet. Leaving it out or counting it as zero would answer a different question.

The model isn't called at all: there's nothing for it to explain. The same goes for a batch that doesn't exist ("There is no batch G in the inspection records you can access") and for a denominator of zero, where a rate isn't defined.

Databases make the same mistake easy: SQL's SUM skips missing values silently, so a query that runs without error can still be computing over incomplete data. Check the inputs before you trust the total.

What changed

PASS Q05-pass-rate found in answerPASS Q18-rate-by-product found in answerPASS Q19-missing-count status cannot_computePASS Q20-definition-stays-search route search; all present

Q05 has been a known gap since Part 2. It's closed, and the check runner has no known gaps left: the remaining modules bring their own questions. The held-out set got a calculation question too ("What percentage of C-3 bikes passed final inspection without rework?"), which the calculate path answers with the same 80.9%.

The system now

flowchart LR
  Q[Question] --> E[Extract product and date]
  E --> R{Router}
  R -- "search" --> S[Search, top k]
  R -- "list" --> L[Every matching record]
  R -- "calculate" --> C["Metric computed in code (records in scope)"]:::new
  K[(Records)] --> L
  K --> C
  C -- "missing input" --> N[Say why there is no result]:::new
  C -- "ok" --> P[Result + definition + records] --> M[Model explains]
  M --> V{{"Numbers check"}}:::new
  S --> M
  L --> M
  classDef new stroke-width:3px

records: 10 error codes, 31 checks, 6 inspection batches (batch F incomplete)metrics: pass_rate

Stop here if

Your numeric questions are about a known set of metrics over records you can extract and keep correct. Each metric is a small, deliberate addition: a formula, the records it reads, the rule for missing inputs, and test cases with known answers.

This module computes over records that already stand on their own. Some questions need evidence that is split across documents that point to each other: a bulletin that changes a procedure in a manual, a part that appears under a different name in each document. Following those links is Part 11.