Caching strategies for LLM systems in finance

The cheapest model call is the one you never make. In a finance LLM system, much of what looks like inference is repetition: the same policy document read into context on every request, the same enrichment recomputed because two services did not share a result. Caching removes that repetition at three layers, and each layer fails differently.

The reason to be careful is that a cache is a correctness surface, not just a cost lever. A wrong answer served from a cache is worse than a wrong answer computed fresh, because it is cheaper and faster to serve, and it comes back the same way on every repeat. In finance the number attached to that answer ends up in a decision, a report, or an audit trail. So the question is never only “will this hit,” it is “what exactly am I keying on, and what makes this entry wrong.”

Prompt caching: the free layer

Prompt caching lives at the provider. You mark a stable prefix of the prompt, and the provider stores the computed attention state for it so the next call with the same prefix skips reprocessing those tokens. In finance prompts the prefix is usually large and static: system instructions, the tool and function schemas, a chunk of policy or a rate card, sometimes a few worked examples. That content does not change between requests, and it is often thousands of tokens. Paying full input price for it on every call is waste.

Two rules make this pay off:

• Order the prompt so everything static sits at the front and everything per-request sits at the back. The cache matches on prefix, so a single variable token near the top invalidates the whole thing. Put the customer, the account, the as-of date, and the retrieved passages at the end.
• Keep the prefix byte-identical. A regenerated timestamp, a reordered JSON key, or a trailing space breaks the match. If you template the system prompt, freeze the template and inject variables only after the cache boundary.

This layer is the one to turn on first because it costs you nothing to build and it does not change what the model sees. The catch is the short time-to-live on the provider side; a cache that expires in a few minutes only helps bursty traffic. For steady load it is close to free money. For a nightly reprocessing job that hits the same prefix a hundred thousand times, it is a large line item.

Semantic caching: fast but sharp

A semantic cache stores answers keyed by the meaning of the input rather than its exact bytes. You embed the incoming request, look for a stored entry within some similarity distance, and if you find one, you return its answer without calling the model. This is where the real latency and cost wins are, and it is also where finance teams cut themselves.

The trap is that similar text does not mean same answer. Ask “what is the outstanding balance on this account” for two different customers and you get one sentence with two different correct answers, and the embedding cannot see the account behind the words. Two KYC questions can be near-identical in phrasing and have opposite correct answers once the subject changes. If the embedding is your whole key, you will serve one customer’s answer to another, which is a leakage incident, not a cache miss.

So the key is never the embedding alone. It is a composite:

• Hard partition first. Tenant, customer or entity id, and the as-of or data-version go into an exact-match component of the key. The semantic match runs only inside that partition. This also keeps one tenant’s traffic from warming another tenant’s cache, which matters for isolation and for the audit story.
• Set the similarity threshold high and treat near-misses as misses. For a marketing FAQ a loose threshold is fine. For anything that produces a figure or a decision, a near-neighbour is a different question. The cost of a false hit here is a wrong number in a report, so bias the threshold toward computing fresh.
• Maintain an eval set of paraphrase pairs and known non-matches, and measure the false-hit rate against it before you widen the threshold. A semantic cache has a false-positive budget exactly like a screening model does, and you should track drift in that rate over time.

Where semantic caching earns its place cleanly is on deterministic, reference-style questions: policy lookups, definitional queries, classifications over a stable taxonomy. On anything that reads live account state, keep the threshold tight or skip this layer and rely on result caching instead.

Result caching: keyed on the data, not the clock

Result caching stores the finished output of a pipeline, keyed on exact inputs. This is the oldest idea and the safest, and in finance it is usually the one that matters most, because so much work is idempotent. Enriching a transaction, resolving an entity, extracting fields from a filing, scoring a document against a policy: run the same inputs through the same pipeline version and you should get the same output. Computing it twice is pure waste.

The key is the whole determinant of the answer, and getting it right is the entire job:

• Content hash of the input, not an id that can be reused. If a document at the same id gets re-uploaded with different bytes, the id-keyed cache serves the old extraction. Hash the content.
• Pipeline version, prompt version, and model id. When you change the prompt or move the model, old results are not valid answers to the new question. Bumping the version invalidates them without a flush.
• The data snapshot or ledger version the computation read. This is the point-in-time discipline that keeps a cache honest. Key on the version of the source, and a restatement or a late-arriving correction changes the key, so a reprocessing run cannot serve figures computed before the correction. Time-to-live alone will not save you here; a five-minute TTL happily returns a stale balance four minutes after a posting.

Result caching also gives you lineage almost for free. Because the key encodes input hash, pipeline version, model, and data snapshot, a cached entry is a record of exactly what produced a given output. When an approver or a reviewer asks why a figure is what it is, the cache key is the start of that answer.

Across all three layers the discipline is the same. Write down what determines a correct answer, put all of it in the key, and let anything you leave out surface as a bug you can name. Caching buys back cost and latency. The key design is what keeps the answers correct, and in finance you earn the first only by getting the second right.