Choosing Your System

Over thirteen parts, the Atlas Bikes assistant grew from a few dozen lines of naive RAG into a system with six routes, records, a link graph, an agent loop, and a comparison engine. Every part started from a question the previous version got wrong and ended with a version that got it right.

The series doesn't ask you to build all thirteen parts. Each checkpoint was a complete system for the questions it answered, and most teams need far less than the last one. This part is about choosing.

Start from the questions

Atlas's dealer desk is a good example. Part 8 gave us a week of their real questions. Which parts of the series do those questions need?

The part-14 checkpoint adds a small tool for exactly that. python -m atlas needs takes a list of questions, runs each through the router, and counts the paths they'd take, with the part of the series that builds each path:

24 questions from traffic/week-1.json
route      questions  built in                   examplesearch       23  95%  Parts 1-6: the core        What battery warranty applies to a C-3 bought on 15 March...list          1   4%  Part 9: every match        Which checks do I need before delivering a T-7?calculate     0   0%  Part 10: calculationsfollow        0   0%  Part 11: linked documentsagent         0   0%  Part 12: an agent loopcompare       0   0%  Part 13: whole documents
Modules these questions need: Part 9: every match

Twenty-three of twenty-four questions need the core. One needs Part 9's list path. For this desk, in this week, the right system is the core plus one module. Calculations, links, an agent loop, and comparisons would be code to maintain with no questions to answer.

The same tool on the test set says every module is needed, which is no surprise: those questions were written to motivate each part. Your users' questions are what count.

You can check the decision before committing to it. Assistant(routes={...}) builds only the modules you choose, and a question routed to a module you didn't build gets the core's search instead. Replaying the week with the core alone brings back Part 8's weak search on r008, the T-7 delivery checks; with the list module, it's gone:

python -m atlas simulate --routes none    ->  Routes: search 24        Weak searches: 1python -m atlas simulate --routes list    ->  Routes: search 23, list 1  Weak searches: 0

Check the routing by hand

The counts come from a router, and in the companion code that router is a few rules. Its counts are a first look, not a verdict.

The held-out questions show why. Run through the same tool, they "need" no comparison module:

compare       0   0%  Part 13: whole documents

Yet one of them, "How does revision 2 of the C-3 battery warranty differ from revision 1?", is a comparison. The stand-in router's rule doesn't recognize that wording, as Part 13 found. A count of zero meant "the router didn't see any", not "there are none".

So read a sample of questions yourself, a few dozen, and label the kind of each by hand. Compare your labels with the router's. And read Part 8's review queue: a question the router sends to search but that search answers badly is a candidate for a module, whatever the count says.

The map

Every part of the series answers one kind of difficulty. Here they are side by side, with the sign that you need each one and what it costs to keep.

The first eight rows are the core and the measurement every system needs, though not all at full strength from day one. Part 1 alone is a complete system if your documents are clean digital text, your questions are single facts, and a person reads each answer with its sources. Parts 2, 7, and 8 grow with the stakes: a handful of cases on day one, a measured release later, a review queue once real users arrive.

The last five rows are modules. Each is independent of the others, and each comes with ongoing work, which is the real cost. Records need owners. Metrics need agreement on their definitions. Links need identifiers someone keeps consistent. An agent loop needs its guards and its budget watched.

Frequency isn't the only measure

One question in twenty-four took the list path. That alone doesn't justify Part 9. What does is the cost of getting it wrong: a list of battery safety checks that comes back short is a safety problem, not an inconvenience. In Part 8, search happened to supply the whole T-7 table; the list path turns that luck into a guarantee.

So weigh each kind of question by two things: how often it comes, and what a wrong answer costs. A recall question may come once a month and still justify Part 12, because answering "affected" for a T-7 sends a customer away with a wrong instruction. A dozen weekly questions about opening hours may not justify anything beyond the core, because a wrong answer costs a phone call.

And when a module isn't worth building, the core can still be honest about what it can't do. A list question answered from search can be made to say that the list may be incomplete. That's often enough until the questions, or the stakes, grow.

How to decide, in order

1. Collect real questions. From logs, support tickets, or by asking the people who will use the assistant. If you have none yet, write them with those people, as in Part 2.
2. Write the expected behavior for a few dozen of them, including what shouldn't happen ("doesn't supply the charger recall"). These become your cases.
3. Label their kinds, by router and by hand. Count them. Note the costly ones.
4. Build the core, measure it on your cases (Part 7), and keep a set of questions you never tune on.
5. Add a module only for a kind of question that needs it, by frequency or by cost. Re-measure. Check that every earlier case still passes.
6. Release, watch, and repeat. The review queue (Part 8) tells you which kind of question to look at next, and the next module, if any, comes from there.

The series followed the same order, with gaps that came from Atlas Bikes. Yours come from your users.

What the series didn't cover

Some topics sit next to this series rather than in it.

Fine-tuning. Training a model on your documents is a different tool for a different job. It's good at teaching a style, a format, or a vocabulary; it's poor at facts that change, can't cite a source, and can't respect who may read what. Company knowledge that changes and must be cited belongs in retrieval; fine-tuning can still help a model use it well.

Agents that act. Every tool in this series reads. Tools that change things, such as orders, records, and messages, need different controls, and they're the subject of a future series, still a work in progress.

Real implementations. The companion code runs on stand-ins so every result is deterministic and runs offline. A version with real models behind the same interfaces is planned, with the same checkpoints, one per part. It will show what the stand-ins can't, especially in the agent loop.

Production concerns. Three companion articles will build on that real implementation, because they're best seen on a real system: access and trust boundaries, including permissions in derived data and prompt injection; keeping the knowledge layer current, through updates, deletions, and migrations; and making the system fast and affordable, including caching.

The system you need

The companion repository has fourteen tags. None of them is the answer. Each is a system that works for a known set of questions and fails on a known set of others, written down in the check runner as known gaps.

For your own system, keep the same record: the questions it answers, the ones it doesn't, and the next gap a user's question has opened.