Long context or retrieval? Choosing for finance documents

Retrieve by default, and reach for long context only when the document is small enough to fit and the question spans the whole of it. A 400k-token filing does not need to sit in the window if the answer lives in two paragraphs; retrieval finds those paragraphs cheaper and with a cleaner audit trail. Long context earns its keep when the reasoning is genuinely global.

The pitch you keep hearing is that context windows got big enough to end the argument. Drop the whole 10-K in, skip the chunking, skip the vector store, let the model sort it out. It is a real capability and it is genuinely useful. It is also the wrong default for most finance document work, and the reasons are boring and operational rather than about model quality.

What each approach actually buys you

The two are not competing at the same layer. Retrieval decides what the model reads. Long context decides how much it can hold at once. You almost always need the first even when you have the second, because in a finance setting you rarely have one document. You have a filing, plus the prior year’s filing, plus the credit agreement, plus internal exposure data, plus three quarters of statements. Retrieval is how you pick which of those to load. The window size only matters after that choice is made.

Where long context genuinely wins:

• The document is a single coherent object and the question is about the whole of it. “Does anything in this 80-page contract conflict with the termination clause?” is a global question. Retrieval fragments it; you cannot retrieve your way to “nothing else contradicts this” because absence is not a chunk.
• Cross-references are dense and local chunking would sever them. Indentures and prospectuses point backward and forward constantly. A chunk that says “as defined in Section 4.2” is useless without 4.2.
• You are doing exploratory reading, not production QA. Analyst-in-the-loop work tolerates the cost; a straight-through processing pipeline running thousands of times a day does not.

Where retrieval wins is the mirror image: many documents, repeated queries against the same corpus, and answers that live in a few identifiable spans. That describes the bulk of what finance operations teams automate.

The cost tradeoff is not close

This is the part people underweight. Say you run a document-QA step inside a reconciliation workflow, a few thousand invocations a day against the same set of loan files. Stuffing 250k tokens of context into every call is not a rounding error. You are paying to have the model re-read a document it read ninety seconds ago for the previous query, and you are paying prefill on tokens that had nothing to do with the question.

Retrieval flips the ratio. Pull the four or five relevant passages, spend 4k tokens instead of 250k, and the per-query bill drops by well over an order of magnitude. Prompt caching narrows the gap for the prefill of a stable document, but it does not close it, and it does nothing for the fact that most of those tokens are irrelevant to the specific question. When volume is high and the corpus is stable, the economics point one direction and it is not subtle.

There is a latency argument too. Long context is slow to first token when the window is full. For an interactive analyst tool that is tolerable. For a pipeline with a quarter-end deadline and a queue behind it, seconds per call compound into hours.

Long context does not mean the model reads everything

The comfortable assumption is that if it fits, it is used. It is not, uniformly. Recall degrades with position and with the number of competing facts in the window, and finance documents are adversarial in exactly this way. A 10-K repeats similar-looking numbers across segments, periods and restatements. The risk is not that the model fails to find a figure. The risk is that it confidently returns the prior-year number, or the segment total instead of the consolidated one, because both were in the window and it grabbed the wrong one. That is a point-in-time correctness failure, and a full context window makes it more likely, not less, by putting every lookalike figure in front of the model at once.

Retrieval, done with decent metadata, gives you a defence here. If each passage carries its period, entity and source, the model has to reason over a smaller, labelled set, and you can trace which span produced the answer. That lineage matters when someone asks why a number landed in a report and you need more than “the model said so.”

None of this is a reason to avoid long context. It is a reason to test it on your own material rather than trust the marketing number. Build a needle test from real filings: plant a known figure at varying depths and measure recall by position across your actual document lengths. Then track it as a first-class eval, because this behaviour drifts across model versions and you do not want to discover a regression at quarter-end.

How we actually decide

The rule we use is close to mechanical. Retrieve unless you can name a specific reason the question needs the whole document held at once. Most cannot, and for those retrieval costs less and leaves a cleaner trail. For the ones that can, use long context deliberately, size the eval to the real document lengths, and watch the false-positive budget because global reasoning over a large window is exactly where confident wrong answers hide.

In practice most finance pipelines end up hybrid anyway. Retrieval narrows a corpus of hundreds of documents down to the one or two that matter, then long context reads those in full when the question is global. That ordering keeps cost bounded and keeps a record of what was read, which is the part an auditor cares about long after the model has moved on.