Measuring Whether It Works

The core is complete: 13 of 15 test cases pass, and the two that fail need modules we haven't built. Can Atlas Bikes put it in front of its staff?

Not yet, for two reasons. First, a passing case says that the right evidence arrived, not how well the system did it. The warranty question passes, but in Part 6 the model also received a table of battery safety checks it had to ignore. How much of what the model reads is noise? Nothing we have measures that.

Second, everything so far ran on nine sample documents. Atlas Bikes has more: a trekking bike (T-7) and a kids' bike (K-1), each with its own warranty, specification, manuals, and error codes; an owner's manual, a customer FAQ, and a brochure covering the same topics in other words; dealer bulletins; a price list only sales may read. The full collection is 29 documents and 114 searchable pieces, instead of 32. A system tuned on a sample has to be measured on the real thing before release.

First, the test set itself breaks

Running the check runner on the full collection gives one regression:

The question was "Which battery check applies when the battery casing has been opened?". With only the C-3, that had one answer. The T-7 has its own check R07, for a 48 V battery, and the question doesn't say which bike. It is now ambiguous. The system didn't get worse; the question did.

The fix belongs in the test set: the case now asks about "the battery casing of a C-3". The same happened, less visibly, to "What does error E-417 mean?": the T-7 has an E-417 entry too, saying the same thing, so it now counts as a correct source. Test questions and their expected answers are written against a collection. When the collection changes, review them.

Measure each stage

For each question, we record which blocks would correctly answer it. Then, comparing that with what search delivered, four numbers describe retrieval:

• Hit: did at least one relevant block arrive? Yes or no.
• Recall: what share of the relevant blocks arrived? If a question needs two blocks and one arrives, recall is 0.5.
• Precision: what share of the arriving blocks are relevant? One relevant block among five is 0.2: the model read four blocks of noise.
• Rank: where the first relevant block appeared. First is better than fifth.

We also estimate the size of each prompt in tokens, because everything the model reads costs time and money.

The code compares two lists of block references, the relevant ones and the supplied ones (measure.py):

supplied = [block.ref for block in trace.context]relevant = set(case.get("relevant", []))found = relevant & set(supplied)if relevant:    hit = 1.0 if found else 0.0    recall = len(found) / len(relevant)    precision = len(found) / len(supplied) if supplied else 0.0    first = next((i for i, ref in enumerate(supplied, start=1) if ref in relevant), None)else:  # nothing in the documents answers it: retrieval can't be scored    hit = recall = precision = first = None

Questions with no relevant block, like the one that needs clarification or the one that needs a calculation, can't be scored this way and are marked as such.

python -m atlas measure prints these numbers per question. On the full collection, with nothing filtering what search returns:

mean recall 1.00   mean precision 0.33   mean prompt ~368 tokens

Every relevant block arrives. Two thirds of what the model reads is noise.

Look at what the noise is

Here is what the 2026 warranty question receives on the full collection:

0.593  WARR-R2-2-p0          the C-3 warranty policy (the answer)0.571  FAQ-CUSTOMERS-2-p0    "Is my battery covered by warranty?"0.450  BROCHURE-C3-2-p0      "Up to 3 years of battery warranty..."0.429  WARR-FRAME-R1-2-p0    frame warranty0.429  WARR-FRAME-R1-3-p0    frame warranty claims

None of it is random. The FAQ answers a related, more general question. The brochure says "up to 3 years", which a model might quote instead of the policy. The frame warranty is a warranty, for a different part. This is what noise looks like in a real collection: plausible neighbors, written for other readers, that a model could mistake for the answer.

Keep questions you never tuned on

An average hides where a system is weak, so each test case is also labeled with its kind: exact code, paraphrase, versioned policy, figure, and so on, and measure reports each kind separately. On our development questions, every kind has recall 1.00, which proves less than it seems to.

Every setting in Parts 1 to 6 was chosen while looking at the same fifteen questions. A system can get good at a test set without getting good at the job, the same way a student can memorize past exam answers.

So before measuring, we wrote held-out questions: new questions, never used while building, many about documents the system was never tested on. They live in a separate file and are scored separately (measure --held-out). The rule that makes them useful: don't change the system to fix a held-out question and keep calling it held out.

The first held-out run found two misses the development questions never showed:

• "Which chargers are affected by the recall?" missed the recall bulletin. The word "recall" appears only in the bulletin's title, and the searched forms from Part 4 included section titles but not document titles.
• "How often should a T-7 be serviced?" missed the service manual, because "serviced" didn't match "service".

The first points to a real gap in the system, so we fixed it: the searched form of every piece now starts with its document title. That question was used to make the fix, so it is no longer held out. It moved into the development cases, and two fresh held-out questions replaced it. One of them, "What must be done before installing firmware 3.2?", missed for the same reason as the second: "installing" didn't match "installation".

Both remaining misses come from our stand-in embedding model, which compares word forms naively; a trained embedding model would match "serviced" and "service". They stay in the held-out set as a reminder that every model has blind spots, and held-out questions are how you find yours. By kind, held-out paraphrase questions have recall 0.75; every other kind has 1.00.

flowchart LR
  D[(Development questions)] --> T{{"Build and tune"}} --> S[System]
  H[(Held-out questions)] --> E{{"Measure only"}}
  S --> E --> F["Estimate for new questions"]
  E -- "a miss you fix" --> D
  E -- "then write new ones" --> H

The document-title fix had a cost, which the same measurement shows. Titles give more pieces something to match, so development precision dropped from 0.46 to 0.37 (with the threshold chosen below). That is the normal pattern: most retrieval changes trade one number against another, and you only see the trade when you measure both.

Let the numbers choose a setting

The noise suggests a simple change: after reranking, drop pieces whose score is below a minimum. Too low and it does nothing; too high and it cuts real evidence. measure --min-score tries a value without changing the system:

The decision is made on the development questions. Recall holds up to 0.5 and falls at 0.6, so we chose 0.4: it improves precision and leaves a margin below the drop. The setting is one line in the retrieval step:

ranked = [(score, c) for score, c in ranked if score >= self.min_score]

The held-out columns then estimate what the choice costs on questions we didn't tune on. One held-out question loses its evidence at any threshold from 0.3: its answer ranked second, with a low score of 0.25. Expect a threshold to cut some correct but low-scoring evidence, here about one question in sixteen. We don't lower the threshold to rescue that question; that would be tuning on the held-out set.

Also notice how little the threshold does on the full collection. On the nine sample documents, the same value lifts precision from 0.43 to 0.68. On the full collection, only from 0.33 to 0.40. The remaining noise is the plausible neighbors from earlier, and they score as high as some real answers. A threshold can't separate them. Better ways exist, such as the product filter from Part 6 applied to more questions, or a stronger reranker, and measurement will tell you whether they help.

Checking answers: judges, and checking the judge

Everything so far measures the evidence. Whether the answer is right is a different question: a model can receive perfect evidence and still state the wrong number.

Checking answers by hand doesn't scale, so teams use a judge: another language model asked something like "Does this answer state only what the sources support?". A judge is a model making judgments, which means it can be wrong. Before trusting its scores, compare its verdicts with a person's on the same answers. This is called calibrating the judge.

Our companion code includes a stand-in judge: a sentence counts as supported when most of its words appear in the sources.

for sentence in re.split(r"(?<=[.!?])\s+", answer.strip()):    words = terms(sentence)    if words and sum(w in source_terms for w in words) / len(words) < self.threshold:        return "wrong"return "correct"

A person wrote six answers and labeled each one correct or wrong. python -m atlas judge-calibration runs the judge on them:

answer  person   judge    noteJ01     correct  correct  Matches WARR-R2.J02     wrong    correct  Wrong number: 2 years is the 2025 policy.  <- disagreeJ03     correct  correct  Matches the E-417 entry.J04     wrong    wrong    That is E-471, the opposite fault.J05     correct  wrong    Correct, in different words from the guide.  <- disagreeJ06     wrong    correct  There are eleven.  <- disagree
Agreement with the person: 3 of 6

The judge agreed half the time. It accepted "2 years" when the right answer is 3, and "eight checks" when there are eleven, because nearly every word matched the sources. It rejected a correct answer written in different words. A score from this judge would look precise and mean little.

A language-model judge is far better than word matching, but it has its own failure modes, such as missing a wrong number in a long answer. The method doesn't change: label a set of answers yourself, include tricky ones like wrong numbers and correct paraphrases, and rely on the judge only for the kinds of answers it gets right.

Where Ragas fits

Ragas is a widely used open-source library for evaluating RAG systems. Its faithfulness metric asks a model to break an answer into claims and checks each against the retrieved contexts: a judge, like ours, only far more capable. Its context precision and context recall metrics measure retrieval quality, in the spirit of our precision and recall.

It needs the same ingredients we already collect: the question, the retrieved contexts, the answer, and a reference. python -m atlas measure --export-ragas eval.jsonl writes our questions with Ragas's field names (user_input, retrieved_contexts, response, reference), so you can load them into Ragas with a real model behind it. Treat its scores like any judge's: calibrate them against your own labels first. See the Ragas metrics documentation.

The system now

flowchart LR
  C[(Development questions)] --> R{{Check runner}}
  C --> M{{"measure: hit, recall, precision, rank, tokens, by kind"}}:::new
  H[(Held-out questions)]:::new --> M
  J[(Answers labeled by a person)]:::new --> K{{"judge-calibration"}}:::new
  A[Assistant + document titles + min score 0.4]:::new --> R
  A --> M
  A --> K
  classDef new stroke-width:3px

Not a stopping point

Measuring is never finished: rerun it whenever documents, settings, or models change. These numbers also come from questions we wrote ourselves. Real users will ask things none of our sets anticipate, and some of their questions will fail in ways no test showed. Part 8 watches the system once those users arrive: what to record for every request, which signals to watch, and how a failure in production becomes the next test question.