Watching It in Production

The assistant passed its checks and its measurements, so Atlas Bikes gives it to the service staff and the sales desk. A week later, one technician has left a thumbs down with a short comment: "Didn't explain the error at all."

Nothing crashed. No error was logged. The answer was a few tidy sentences with citations, like every other answer. A RAG system fails quietly: the wrong answer looks just like a right one. To find out what happened, and how often it happens without anyone complaining, the system has to keep a record of every request.

This part adds that record, then reads one week of it.

What to record for every request

Each request goes through the same stages: extract the product and date, search three ways, fuse, rerank, check for missing details, assemble the prompt, call the model, resolve the citations. When an answer is wrong, the question is always which stage went wrong. So we record each stage separately.

A record of one stage is called a span: its name, how long it took, and the few details that explain its result. For the exact lookup, that's the codes it recognized and the entries it found. For the reranker, the best score and how many pieces fell below the minimum. All the spans of one request together form a trace.

In the companion code, a span is a small context manager. It times the stage and adds the details the stage writes into it:

@contextmanagerdef span(trace, stage: str, **details):    """Time one stage and keep its details on the trace."""    record = {"stage": stage, **details}    start = time.perf_counter()    try:        yield record    finally:        record["ms"] = round((time.perf_counter() - start) * 1000, 3)        if trace is not None:            trace.spans.append(record)

Each stage of the pipeline is wrapped in one:

with span(trace, "search.exact") as s:    codes = error_codes(question)    exact = self.index.lookup("error_code", codes, groups, where)    s.update(codes=sorted(codes), found=[c.id for c in exact])

After the answer, the assistant appends one line to a log file (one JSON object per line, a format called JSON Lines). The line holds the question, the outcome, what was retrieved with its scores, which blocks the model received, the prompt size, the answer, the spans, and the settings in use: the checkpoint, k, the minimum score, and which models. Without the settings, a request from last month can't be explained once the system has changed.

{"id": "r001", "at": "2026-10-19T08:12:00", "user": "u-10de72a2", "groups": ["employees", "staff"], "question": "What battery warranty applies to a C-3 bought on 15 March 2026?", "status": "answered", "top_score": 0.593, "blocks": ["WARR-R2#2", "FAQ-CUSTOMERS#2", "BROCHURE-C3#2", "WARR-FRAME-R1#2", "WARR-FRAME-R1#3"], "prompt_tokens": 331, "spans": ["..."], "settings": {"checkpoint": "part-08", "k": 5, "min_score": 0.4, "...": "..."}}

Feedback arrives later, when the user clicks a thumb, so it is written as a separate event that points to the request's id.

A week of traffic

The companion code can't have real users, so it has a simulated week instead: 24 questions from service staff and the sales desk, in their own words, with the feedback some of them left. They were written without looking at what the system would do with them. Some are typed carelessly, some ask about products that don't exist, some repeat.

python -m atlas simulate replays them through the assistant with logging on. python -m atlas stats then summarizes the log. (These outputs come from the system as it was during the week, git checkout part-08~1; timings differ slightly on every run.)

24 requests from 2 users, 2026-10-19 to 2026-10-24
Outcomes:  answered      22  (91%)  clarify        2  (8%)  answered with nothing supplied: 0
Weak searches (best score < 0.5, no exact match): 1Answers withheld for unknown citation labels:     0Feedback: 4 up, 2 down, 18 requests without feedback
Time per stage, milliseconds (median / 95th percentile):  total              1.748 /    2.262  search.keyword     0.702 /    1.057  rerank             0.525 /    0.679  search.vector      0.352 /    0.447  ...Prompt size: ~296 tokens on average, 489 at most
Products named in questions: none named 10, T-7 5, C-3 4, E-417 2, C-5 1, K-1 1, E-447 1Asked 2 times: 'What does E417 mean?'...

Each line answers a question an operator would ask:

• Outcomes. How often the assistant answers, asks for details, or withholds an answer. A jump in clarifications after a release often means extraction broke.
• Weak searches. Requests where no exact match was found and the best reranked score was below 0.5. The assistant may have answered from poor evidence.
• Feedback. Two thumbs down out of six ratings. Note the 18 requests without feedback: most users don't rate anything.
• Time per stage. The median and the 95th percentile (p95, the time 95% of requests stay under). The p95 shows the slow requests a median hides.
• Prompt size. Tokens are cost. A slow rise means the assembled context is growing.
• Products named. What users ask about. Here it also shows a problem: "E-417" and "C-5" were extracted as products. Validation from Part 6 drops them before they filter anything, but the extractor is confused, and now we can see it.
• Repeated questions. A question asked twice may have failed the first time.

Following one request

The log also builds a review queue: requests a person should look at. A request joins the queue when the user gave a thumbs down, the assistant asked for clarification or withheld its answer, nothing was supplied, or the search was weak.

Review queue:  r002  asked for clarification                      battery warranty  r007  thumbs down                                  What does E417 mean?  r008  weak search (0.406)                          Which checks do I need before delivering a T-7?  r010  thumbs down, asked for clarification         How long does the battery warranty last on a C-5 bought in May 2026?

r007 is the complaint from the opening. python -m atlas trace r007 shows it stage by stage:

r007  2026-10-20T09:15:00  u-10de72a2 (employees, staff)Question: What does E417 mean?Status:   answeredFeedback: down - Didn't explain the error at all
Stages:  extract           0.013 ms  product=None, on_date=None  search.exact      0.015 ms  codes=[], found=[]  search.keyword    0.622 ms  top=['TRAIN-C3-2-search', 'MANUAL-C3-OWNER-3-p0', 'FAQ-CUSTOMERS-5-p0']  search.vector     0.339 ms  top=['TRAIN-C3-2-search', 'FAQ-CUSTOMERS-5-p0', 'MANUAL-C3-OWNER-3-p0']  fusion            0.009 ms  shortlist=10  rerank            0.441 ms  top_score=1.0, dropped_below_min=6  clarify-check     0.002 ms  missing=[]  assemble          0.013 ms  blocks=['TRAIN-C3#2', 'MANUAL-C3-OWNER#3', 'FAQ-CUSTOMERS#5', 'TRAIN-T7#2'], prompt_tokens=321  ...

Read it top to bottom:

1. extract: no product, no date. Fine; the question doesn't name one.
2. search.exact: codes=[]. The user typed "E417" without the dash. The exact lookup from Part 5 only recognizes codes written the catalog's way, "E-417", so it found nothing.
3. search.keyword and search.vector: without the code, both searches matched the remaining words, "what does … mean", and found passages about the display and its blink patterns.
4. rerank: the best score is 1.0. The reranker was fully confident, because those passages contain every remaining word of the question.
5. assemble: the model received four blocks about the display. The E-417 entries never reached it.

The failure happened at step 2, and every later stage worked correctly on what it was given. Without the spans, all we would know is that the answer was bad. With them, the cause takes a minute to find.

The trace also shows why the automatic signals didn't catch it. The search wasn't weak: its top score was 1.0. A confident score from a stage that received the wrong input looks exactly like a correct one.

What the signals missed

Only the thumbs down caught r007. The repeated questions line holds the rest of the story: "What does E417 mean?" was asked twice. r020, three days later, got the same display passages. That user left no feedback, and nothing in the review queue points to it. For every complaint, expect several silent failures.

r006 is worse, because no signal touches it. A technician asked for the dealer price of a T-7 battery. The price list is readable only by sales, so access control did its job: the price list never reached the model. Then the assistant answered anyway, from four T-7 service and specification blocks that mention batteries but not prices, scored at 0.6. Nothing about it looks wrong: an answer, citations, a reasonable score, no feedback. For comparison, r005 asked the same question from the sales desk and received the price list first.

The only way to find failures like r006 is to read a sample of ordinary requests as well as the flagged ones. A few per day, chosen at random, read by someone who knows the documents. It's slow, and nothing replaces it: the review queue finds the failures you anticipated, and sampling finds the ones you didn't.

Signals are hints, not verdicts

Two other requests in the queue show the opposite problem.

r008 was flagged as a weak search, with a best score of 0.406. Its trace shows the model received one block, the T-7 table of battery checks before delivery, which is exactly the right answer. The searched pieces were single rows of that table, and one row shares only a few words with the question, so every score was low. Assembled, the rows became the whole table. A false alarm.

r010 asked how long the battery warranty lasts on "a C-5 bought in May 2026" and left a thumbs down: "It asked me which product, I told it the C-5." The trace shows why. Atlas Bikes doesn't sell a C-5, so validation dropped it. The stand-in extractor only reads full dates, so "May 2026" was missed too. The assistant asked for both. Its behavior followed the rules from Part 6, but the right response is different: say that there's no C-5 and name the models that exist.

So a person decides what each flagged request means. Each one ends with one of three decisions: no action (r008), accepted but not fixed yet (r010, r006), or fix now (r007).

From a failure to a test case

A failure seen in production should never be able to return silently. So it becomes a test case, and the check runner from Part 2 guards it from then on.

python -m atlas promote r007 copies the request into a draft file, cases/candidates.json, with what was observed:

{  "id": "P-r007",  "from_request": "r007",  "question": "What does E417 mean?",  "observed": {    "status": "answered",    "supplied": ["TRAIN-C3#2", "MANUAL-C3-OWNER#3", "FAQ-CUSTOMERS#5", "TRAIN-T7#2"],    "why_flagged": ["thumbs down"]  },  "behavior": "TODO: write the expected behavior from the source documents",  "checks": []}

The draft can't become a test case on its own: the log records what happened, and only a person can say what should have happened. Here, the E-417 entries must reach the model. Completed, it moves into the test set as Q14-code-without-dash. Running the check runner then shows a regression, as it should, because the system as released can't pass it yet.

The fix follows the trace to step 2. Codes are now recognized with or without the dash and written back in the catalog's form:

def error_codes(question: str) -> set[str]:    """Codes as the catalog writes them. Users also type "E417"."""    return {f"{prefix}-{number}" for prefix, number in re.findall(r"\b([A-Z]+)-?([A-Z]?\d+)\b", question)}

The same request, replayed after the fix:

  search.exact      0.036 ms  codes=['E-417'], found=['ERR-C3-R2-4.17-p0', 'ERR-T7-R1-5.2-p0']  ...  assemble          0.015 ms  blocks=['ERR-C3-R2#4.17', 'ERR-T7-R1#5.2', 'TRAIN-C3#2', 'MANUAL-C3-OWNER#3', 'FAQ-CUSTOMERS#5'], prompt_tokens=343

r020 gets the same correction. The check runner:

r010 and r006 stay in candidates.json with their expected behavior written down: one needs the assistant to recognize a product that doesn't exist, the other needs it to say when the evidence it may read doesn't answer the question. Neither has checks yet. They wait there until someone decides they're worth the work, and the next week's log shows whether they come back.

In the companion repository, the fix is a separate commit after the observability tooling. git checkout part-08~1 gives the system as it was during the week, so you can reproduce the failure and its trace yourself.

The system now

flowchart LR
  U[Staff and sales] --> A[Assistant]
  A -- "spans per stage" --> L[(Request log, JSON Lines)]:::new
  U -- "thumbs up / down" --> L
  L --> S{{"stats: outcomes, weak searches, feedback, time per stage"}}:::new
  S --> Q[Review queue + random sample]:::new
  Q --> P[Person reviews a trace]:::new
  P -- "promote, complete" --> C[(Test cases)]
  C --> R{{Check runner}}
  R --> A
  classDef new stroke-width:3px

Not a stopping point

Watching never ends: keep logging, keep reading a sample, and keep promoting confirmed failures. The log will also tell you which gaps matter. If users keep asking "which bikes are affected by the recall" and get partial lists, or ask for numbers the documents only let you compute, those are signs you need one of the modules that follow. Parts 9 to 13 are independent: an intermediate knowledge base for complete lists, calculations, linked documents, an agent loop, and whole documents. Add only the ones your log says you need.