There is a demo that everyone has seen. You drop a clean PDF into an LLM, ask for the revenue figure and the renewal date, and it comes back correct. It feels solved. Then someone points the same prompt at three thousand filings and a folder of scanned contracts, and the accuracy you measured on one document disappears.
That gap is the whole problem. Document extraction in finance is rarely about the clean case. It is about the 10-K with a footnote that overrides the headline number, the contract scanned at an angle, the table that spans a page break, and the counterparty whose name is spelled four different ways. The unstructured data you actually receive is not the unstructured data in the demo.
Why one good PDF tells you nothing
A single document hides the failure modes that matter at scale. Layouts are inconsistent across issuers and across years. Scanned contracts arrive as images, so before any extraction you are dealing with OCR quality and reading order. The field you want sits in a table on page 40, defined by a clause on page 12, and a naive text dump flattens both into a soup the model cannot reason over.
So we do not treat filings extraction as one model call. We treat it as a pipeline:
- Layout-aware parsing first, so tables, headers and reading order survive instead of collapsing into a flat string
- Retrieval over the parsed document, so the model sees the right page and clause rather than the whole filing
- An LLM to read the retrieved context and return structured fields
- Validation of every field against its expected type and range
- Routing of low-confidence results to a person before anything is trusted
The parsing and retrieval steps are what make contract data extraction survive real layouts. The LLM is doing the easy part, reading a short, relevant passage. The hard engineering sits on either side of it.
Validate the value, not just the format
A returned number that parses cleanly can still be wrong. So extracted fields get checked against what they are supposed to be. An effective date has to be a date. A notional has to be positive and inside a plausible band. A reporting period has to be consistent with the document it came from. When a value fails these checks, that is signal, and it feeds the confidence the system assigns to its own output.
Confidence is where the human comes in. A blanket “send everything to review” defeats the purpose, and “trust everything” is how a bad number reaches a report. Between those, a threshold. Fields the system is confident about and that pass validation flow through. Fields below the threshold, or that fail a check, route to a reviewer with the document open to the relevant page. The reviewer corrects the value, and that correction becomes a labelled example for the eval set.
Every value points back to its source
The piece we never drop is the citation. Each extracted value carries a pointer back to the page and the clause it came from. A revenue figure links to the table that contained it. A termination right links to the paragraph that grants it.
This is what makes the output auditable rather than merely plausible. When someone asks where a number came from, the answer is a location in a document, not a confident sentence from a model. It is the same standard we hold for any financial data: an answer you cannot trace is an answer you cannot put in front of an auditor. Extraction earns its place in a finance workflow only when every field it produces can be checked back to the line it was read from.