From Raw Files to Structured Documents

At the end of Part 2, our check runner listed three questions the assistant can't answer yet:

Here is what the model receives for the first one, "Is a rear mudguard required on the cargo version of the C-3?":

Source: ASSEMBLY-C3-R3, section 4.2Exception: on the cargo version of the C-3, the rear mudguard is not requlredbecause the cargo rack covers the wheel. Keep the rack dra1nage holes clear.
Source: SPEC-C3-R5, section 5The C-3 frame is aluminum with an internal battery tube. The cargo version addsa reinforced rear rack that covers the top of the rear wheel.Page 12 of 40
Source: SPEC-C3-R5, sectionAtlas Bikes | C-3 Specification rev. 5 | Internal
Source: SPEC-C3-R5, section 5Atlas Bikes | C-3 Specification rev. 5 | Internal

The assembly guide is an old scanned PDF. The scanner read "required" as "requlred" and "drainage" as "dra1nage". A capable model might guess its way through the typos, but a keyword search for "required" won't match them. Below that, two of the five passages are nothing but a page header, repeated on every page of the specification. They took the place of evidence.

The other two failures are worse because nothing visible goes wrong. The meaning of two blinks on the display is written in a training slide's speaker notes, and the capacity of battery pack A is printed only inside a bar chart. Our assistant never saw either.

In all three cases, the information was damaged or dropped before it was indexed. In many real projects, this layer, going from raw files to something a system can use, takes more effort than everything after it.

What a parser gives you

Nobody reads a PDF directly. A document parser opens the file and reports what it found on each page. Layout-aware parsers go further than pulling out text: they label each piece as a heading, paragraph, table, figure, or page header, and keep its page number.

Our corpus stores that parser output. Here is part of the C-3 specification, where the battery safety checks table starts on page 13 and continues on page 14:

{"id": "e8", "type": "table", "page": 13, "header": ["ID", "Check"], "rows": [["R01", "Check that the battery serial number matches the delivery note."], ...]},{"id": "e9", "type": "page_footer", "page": 13, "text": "Page 13 of 40"},{"id": "e10", "type": "page_header", "page": 14, "text": "Atlas Bikes | C-3 Specification rev. 5 | Internal"},{"id": "e11", "type": "table", "page": 14, "header": null, "rows": [["R06", "Check that the battery lock engages and the key turns freely."], ...]}

The parser did its job. It found two tables because the page break cut one table in two, and the second part has no header row. It also found a footer and a header between them. Deciding that these are one table, and that the footer and header are noise, is our job.

In Part 1, the assistant ignored all these labels. It took every piece of text in order and joined it into one long string, which is what most first prototypes do. That shortcut loses five things:

• Page headers and footers become part of the text, on every page.
• The split table becomes loose lines, and its scope ("C-3, 36 V battery pack, frame-mounted, standard charger") drifts away from its rows.
• OCR mistakes go straight into the index.
• Speaker notes are never extracted, because only the slide text is taken.
• The figure keeps its caption and loses its values, which exist only in the image.

Convert, then check

The fix is a conversion step between the parser and the index. It turns parser output into the document a reader would recognize and reports what it changed:

flowchart LR
  P[Parser output] --> C{{Convert}}
  C --> S[Structured documents]
  C --> R[Ingestion report]
  R --> H[Person reviews flagged items]
  H --> F[(corrections.json)]
  F --> C
  S --> I[(Index)]

In the companion code this is convert.py. The important parts are short.

Drop page furniture. Headers and footers repeat on every page and carry no evidence:

if kind in ("page_header", "page_footer"):    report.removed_furniture += 1    continue

Rejoin split tables. A table with no header that directly follows another table is its continuation:

if raw["header"] is None and previous and previous.kind == "table":    previous.data["rows"].extend(raw["rows"])    report.merged_tables += 1

The same step attaches the scope paragraph printed above the table to the table itself, so the scope can't drift away later.

Flag uncertain OCR instead of trusting it. Our parser gives each scanned element a confidence score. Low scores go into the report for a person to check:

if raw["id"] in fixes:    text = fixes[raw["id"]]    report.corrected.append(f"{raw['id']} (page {raw.get('page')})")elif raw.get("ocr_confidence", 1.0) < LOW_CONFIDENCE:    report.needs_review.append(...)

Turn figures into text. A vision model looks at the chart and writes a description that keeps its values. The companion code uses a stand-in that reads a prepared description, so everything runs offline:

description = describer.describe(raw["image"], raw["caption"])

Keep slide notes. Each slide becomes its own section with its title, text, and speaker notes.

Every element keeps its type. This article still renders everything to plain text for the old chunker. Part 4 uses the types directly.

Read the report before you trust the index

Conversion produces an ingestion report. Run it before anyone has reviewed the scanned guide:

$ python -m atlas ingest-report --no-corrections
ASSEMBLY-C3-R3 (c3-assembly-guide-r3-scan.pdf)  - NEEDS REVIEW: e5 (page 9) OCR confidence 0.61: Exception: on the cargo version of the C-3, the rear mudguar...SPEC-C3-R5 (c3-specification-r5.pdf)  - removed 6 page headers/footers  - rejoined 1 table split across pagesTEST-BAT-R1 (battery-cycle-test-r1.pdf)  - removed 2 page headers/footers  - described 1 figure(s)

The report tells a person where to look. Someone opens page 9 of the scan, reads the paragraph, and records the correct text in a corrections file, keyed by the parser's element ID:

"ASSEMBLY-C3-R3": {  "e5": "Exception: on the cargo version of the C-3, the rear mudguard is not required because the cargo rack covers the wheel. Keep the rack drainage holes clear."}

Keying the fix to the element ID means it survives the next time the documents are ingested. A fix made directly in the index would be lost on the next rebuild.

The report also checks what code can check without a person. Numbered rows such as R01 to R11 should be consecutive, so a gap is flagged. Had the parser dropped R07, the report would say so before anyone asked about battery safety.

The figure description deserves the same caution. A vision model can misread a bar, so spot-check its numbers against the image, at least for documents people rely on.

The system now

flowchart LR
  subgraph prepare["Prepare documents"]
    P[Parser output] --> C{{Convert + report}}:::new --> T[Text] --> CH[Size-based chunks] --> I[(Index)]
  end
  subgraph answer["Answer a question (unchanged)"]
    Q[Question] --> S[Search] --> PR[Prompt] --> M[Model]
  end
  I -.-> S
  classDef new stroke-width:3px

The three cases now pass, with no regressions:

The figure question now receives the values:

Source: TEST-BAT-R1, section 3Figure description: Horizontal bar chart. Pack A keeps 82 percent of its originalcapacity after 500 charge cycles. Pack B keeps 91 percent of its original capacityafter 500 charge cycles.

The blink question now receives the slide with its notes as its first passage: "Two blinks followed by a pause means the speed sensor signal is lost."

The questions that already worked, the 2026 warranty and the forgotten PIN, still pass. Nothing in how questions are answered changed; only what the index contains.

Other formats you will meet

Our corpus covers a digital PDF, a scan, a slide deck, an HTML page, and Word files. Real collections add more, and each has its own trap.

Stop here if

Stop here if your documents are mostly prose with simple layout, and your questions don't depend on tables, exceptions, or figures staying intact. Keep the ingestion report running on every update either way.

Otherwise, one of our earlier questions still fails:

Asked "Do I need to fit a rear mudguard on a C-3?", the assistant receives the rule as its top passage and the cargo exception as its third. The text is now correct, but the chunker cut section 4.2 between the rule and its exception. Part 4 deals with what we search and what we hand the model, and why they shouldn't always be the same piece of text.