A pilot that costs a few hundred dollars a month tells you almost nothing about what the same system costs in production. The demo runs a handful of queries a day against a clean prompt. Production runs thousands, against context that has quietly grown, with retries you forgot were there. The bill arrives at quarter-end and someone in finance asks the only question that matters: what does one query cost, and why is it that number.
If you cannot answer that, you do not have LLM cost control. You have a monthly surprise. And in finance an unexplained line item is not just awkward, it is the kind of thing that gets a project paused while everyone hunts for the cause.
Where the money actually goes
The headline price per million tokens is the part everyone reads and the part that matters least. The cost lives in how many tokens you push through, how often, and how much of it you are paying for twice.
In practice the bill is driven by a few specific things:
- Oversized context. Stuffing the whole document, the whole history and a generous system prompt into every call because trimming felt risky. You pay for every token on every call, including the ones the model never needed.
- Over-calling. Chains that make five model calls where two would do, or an agent that re-asks the same sub-question because nothing remembered the answer.
- Retries. A timeout, a malformed JSON response, a rate-limit backoff. Each retry is a full second call at full price, and retries spike exactly when the system is under load and you are least watching.
- Output tokens. Usually priced higher than input, and the easiest to let run long when nobody constrained the response.
None of these show up as a single big number. They show up as a per-query cost that is two or three times what the architecture diagram implied.
Methods that move the bill
Once you know where the money goes, the fixes are concrete and mostly boring.
Prompt caching is the first lever and often the largest. If a long system prompt or a fixed instruction block repeats across calls, caching it means you stop paying full input price for the same tokens every time. The same logic applies to retrieval: cache the retrieved passages for repeated questions rather than re-embedding and re-fetching on every hit. Caching only helps where the prefix is genuinely stable, so it is worth structuring prompts so the stable part comes first.
Batching is the second. Many finance workloads are not interactive. Filings extraction across a universe, overnight reconciliation, a backfill of reason codes for an adverse action review. These do not need a sub-second answer, so they can go through a batch path that providers price well below the live one. Sorting work into “a human is waiting” and “this runs overnight” is usually the single biggest structural saving available.
Model routing is the third. Not every sub-task needs the largest model. Entity resolution, a yes/no classification, a short rewrite, a first-pass filter before the expensive reasoning step. Route those to a smaller, cheaper model and reserve the large one for the work that actually needs it. The risk is quality drift on the routed-down tasks, so each route needs its own eval set and a check that accuracy holds. Cheaper is only cheaper if the answer is still right.
Then there is context itself. Better retrieval is a cost method, not only a quality one. Tighter retrieval that returns three relevant passages instead of twenty cuts input tokens on every call and tends to improve the answer at the same time. Trimming context is where good retrieval pays for itself twice.
Make cost a first-class metric
The thing that holds all of this together is measurement. Cost per call has to be a metric you record next to latency and accuracy, broken down by route, by feature and by token type. Without it, every saving above is a guess and every regression is invisible until the bill lands.
Treat it the way you would treat any number that ends up in front of an auditor. You want lineage on the cost: which model, how many input and output tokens, cache hit or miss, how many retries. When finance asks why this quarter’s per-query cost moved, the answer should be a query, not a meeting.
That is the standard worth holding the system to. A per-query cost you can read, explain and defend will survive a CFO’s questions. A monthly total you cannot break down will not.