Before You Retrieve: Turning Documents Into a Corpus

Three parts in, and every one of them has quietly assumed the same thing. Time to admit it.

They assumed the corpus exists — that your documents are already sitting in an index, in a form a retriever can actually find. For a plain-text wiki, fine. For most real documentation, that assumption is where the project quietly fails, weeks before anyone blames the retriever.

Because the index is not your documents. It's what you chose to put in it, in the shape you chose to put it. And the retriever can only ever find what ingestion made findable.

This is the upstream half of RAG. It decides more of your quality than the choice of vector database, and it gets a fraction of the attention. So this part is about the pipeline that runs before the first embedding: parsing, extraction, chunking, enrichment. The unglamorous work where the answers are won or lost.

Parsing: getting the text out at all

Start with the most basic failure, the one nobody demos.

A PDF is not text. It's a description of marks on a page. Pull the text out naively and a two-column layout interleaves into nonsense, a table becomes a run-on of numbers, a header floats into the middle of a paragraph. The reading order the human eye reconstructs effortlessly is simply not in the byte stream.

And that's the easy case, where the PDF has a text layer at all. Scanned documents — which is most of what an established company actually has — are images. No text to extract. You need OCR before anything else, and OCR makes its own errors, on exactly the identifiers (part numbers, codes) that lexical search is best at finding.

None of this is visible in a prototype built on clean markdown. All of it surfaces the day you point the pipeline at the real archive.

The rule: inspect what your parser actually produces, before you index a single document. Layout-aware parsing — the kind that preserves reading order and recognizes structural regions — is not a nice-to-have on a real corpus. It's the floor.

Tables and structure

The KPI system from part three failed on exactly one thing: it treated tables as text.

Chunk a spreadsheet or an inline table as prose and you get fragments of rows severed from their headers. "94.2" ends up in one chunk, "first-pass yield, line 3, Q2" in another, and no retriever reunites them, because the thing that connected them was a grid, and the grid is gone.

Tables carry meaning in two dimensions. Flatten them to one and the meaning doesn't survive.

So tables get handled on purpose. Extract them into an actual structured store, where a KPI question becomes a query. Or, when they must stay in the text index, serialize them in a form that keeps rows intact — markdown tables, or one row per record with its headers repeated — so a chunk is at least self-contained. Which path you choose comes straight from the golden set: if people ask for computed numbers, the table belongs in a structured store, not an embedding.

This is the substrate dimension again, doing its quiet work: the highest-leverage move is often changing what shape the data is in before any retrieval touches it.

The parts that aren't text

Now the part most pipelines ignore entirely.

Technical documentation is full of things that aren't prose: diagrams, schematics, annotated photos, charts, wiring layouts. In an industrial corpus these often are the content — the paragraph next to the diagram just says "see Figure 4." Index the paragraph, drop the figure, and you've indexed a pointer to the answer instead of the answer.

The technique that works is to give each non-text element its own treatment at ingestion. Run the image through a vision model, generate a description of what it shows, and attach that description to the chunk the image belongs to — so the diagram becomes searchable through its caption. Better still, store the image as its own retrievable object, with the generated description as its searchable text and a pointer back to the original, so an answer can surface the actual figure and not just talk around it.

The alternative is native multimodal embedding — putting images and text in one shared vector space and retrieving across both. Cleaner in principle, and improving fast, but describe-then-embed has a real advantage today: the description is inspectable. You can read what the model thinks the diagram shows, and fix it, in a way you can't audit a raw image embedding.

Either way, the point stands: a corpus is not only its words. Deciding what happens to everything else is ingestion work, and skipping it silently caps the ceiling on any question whose answer lives in a picture.

Chunking is a design decision, not a setting

Somewhere in most tutorials there's a line like chunk_size=512, overlap=50, presented as a default.

It's not a default. It's the single granularity decision from part one, and picking it by character count is picking it by the one criterion that has nothing to do with meaning.

The failure is mechanical. Split on a fixed character count and you cut through the middle of a table, sever a conclusion from the premise it rests on, or orphan a clause from the "notwithstanding the above" it depends on. Each resulting chunk is individually retrievable and individually meaningless, and no ranking algorithm recovers meaning that was destroyed before indexing.

Better strategies chunk on structure instead of length: on headings, on paragraph and section boundaries, on the document's own outline. And three patterns are worth naming, because they keep coming up:

• Parent-child, or small-to-big. Index small, precise chunks so retrieval is sharp — but when one hits, return its larger parent section so the model gets enough surrounding context to actually use it. You search narrow and read wide.
• Contextual enrichment. Before embedding a chunk, prepend a short description of where it sits in its document — which spec, which section, what it's about. Anthropic's contextual retrieval reports this meaningfully cuts retrieval failures, and the reason is intuitive: a chunk that says "the limit is 4mm" is useless out of context; "In spec 4-112, corrosion protection: the limit is 4mm" is findable.
• Index the questions, not the chunk. At ingestion, have a model generate the questions each chunk can answer, and embed those — so a user's question matches a question instead of a passage. It's the mirror image of HyDE from part three: HyDE fabricates a document from the query at search time; this fabricates queries from the document at index time, paying the cost once instead of on every request. It travels under a few names — reverse-HyDE, hypothetical question embeddings — and it shines exactly where question and source share no vocabulary.

The rule that survives all of it: if retrieval quality disappoints, look at your chunks before you look at your database. It's the cheaper fix, and usually the real one.

Metadata is retrieval infrastructure

The last thing ingestion produces isn't content. It's the labels on the content.

Every chunk can carry structured fields extracted at ingestion time: source document, section, date, author, version, document type — and, critically, whatever governs who's allowed to see it. This metadata is not decoration. It's what makes whole categories of question cheap later.

A filtered question — "requirements in the current revision of spec 4-112" — is a metadata filter, exact and complete, not a similarity search you hope lands. Freshness questions need dates. And permission-scoped retrieval, which we'll come to, is only possible if each chunk knows which access tier it belongs to. None of that exists unless ingestion put it there.

Extracting good metadata is real work — often its own model-assisted pass. It's also the work that turns a pile of embedded text into something you can query with precision instead of vibes.

Ingestion is a pipeline, not a script

One last reframe, because it changes how you resource this.

Ingestion is not a one-time load you run once and forget. Sources change, and stale chunks have to be re-parsed and re-indexed. Embedding models get upgraded, and the whole corpus must be re-embedded to stay in one vector space. Every derived layer — extracted tables, image descriptions, metadata — inherits the extraction model's error rate and blind spots, and needs the same review discipline you'd give any generated content.

It is, in other words, a production system with its own failure modes, its own cost curve, and its own maintenance burden. Budget it that way. The teams that treat ingestion as a preprocessing afterthought are the ones debugging "bad retrieval" six months later, when the retriever was fine all along — it was faithfully finding exactly the garbage that ingestion put in.

The retriever gets the credit and the blame. But the corpus was decided up here.

---

Two things remain before we can choose an architecture. The corpus is now in a state worth searching. But every one of the seven families raises the same question, and it's been hanging since part three: given all these instruments, who picks which one runs for a given question?

For most of RAG's history, a developer did, in advance. Increasingly, the model does, at request time. That shift — agentic retrieval — is part five.