LLM observability and tracing for finance workloads

A finance LLM system is diagnosable when any answer it produced last quarter can be reconstructed exactly: the resolved prompt, the retrieved context and its versions, the raw output, the token counts, and the model string. Observability is the discipline of capturing that trace on every call, then running cheap checks against it in production so a regression surfaces as an alert instead of a customer complaint.

Most teams instrument the wrong layer first. They add a dashboard of request volume and average latency, which tells you the service is up but nothing about whether the answers are correct. In a finance workload the failure that hurts you is a confident sentence with a wrong number in it, and that failure is invisible to infrastructure metrics. It shows up only if you captured what the model saw and what it said, and then compared the two.

A trace is a tree, and that tree is your unit of work

A single user question can fan out into a retrieval step, three tool calls, a re-ranking pass, and two model completions. If you log those as separate lines, you cannot answer the one question that matters when something breaks: what did the whole pipeline do for this specific input. So the span you care about is the trace, a tree that ties every sub-call back to one root, with a stable ID you can hand to an analyst or paste into a ticket.

For each node in that tree we capture:

• The resolved prompt, after template rendering and context injection, not the template. The template tells you the intent; the resolved prompt tells you what actually went to the model, including the retrieved figures that may have been wrong.
• Retrieved context with document IDs and version or as-of dates. Point-in-time correctness lives here. If your retriever pulled a restated 10-Q that did not exist at the decision date, the answer looks defensible and is quietly leaking future information.
• Raw model output before any parsing. Parsers hide failures. When JSON extraction falls back to a default, you want to see the malformed output that caused it.
• Token counts for input and output, the model and version string, temperature, and latency per node.

Two practical rules. First, propagate the trace ID through every hop, including tool calls that leave your process, or the tree loses branches. Second, sample nothing on the write path in finance. Storage is cheaper than an unreproducible reconciliation break at quarter-end. Sample later, when you decide what to grade.

Token counts are your earliest drift signal

Teams treat token counts as a billing line. They are also the cleanest early-warning signal you have for behavioural drift. A retrieval bug that starts stuffing twice as much context into the prompt shows up as a step change in input tokens days before anyone notices the answers got vaguer. A prompt-injection attempt that makes the model ramble shows up as an output-token spike on a narrow set of traces.

So we account tokens per trace, per prompt template, and per tenant, and we alert on distribution shifts rather than totals:

• Input tokens per template. A jump means your context assembly changed, usually because a retriever returned more or longer documents than the prompt was designed for.
• Output tokens per template. A jump on a task that should produce a fixed schema means the model stopped respecting the format, often the first symptom of a model version change on the provider side.
• Cost attributed to the tenant and the workflow, so straight-through processing rates can be read against spend. A workflow that needs three retries per success is telling you something the average latency chart will not.

The point is that token telemetry, captured at the trace level, doubles as a behavioural monitor. You already pay to count tokens for the invoice. Route the same numbers into your alerting.

Evaluation has to run in production too

Offline eval sets catch what you thought to test. Production catches the rest. The gap matters more in finance than in general chat, because your inputs are adversarial by nature: filings get restated, entity names collide across subsidiaries, and a counterparty that was one legal entity last month is two after a carve-out. An offline set frozen in March will not contain June’s edge cases.

We run two tiers of checks continuously. Reference-free checks run on every request and need no labels:

• Schema and type validation on structured output, so a malformed number never reaches straight-through processing.
• Numeric grounding: every figure in the answer must trace to a value in the retrieved context, or the answer is flagged. This is the single highest-value check for finance, because it catches the confident-wrong-number failure directly.
• Citation coverage and refusal rate, tracked as time series. A rising refusal rate after a deploy usually means a prompt change made the model over-cautious.

The second tier is graded evaluation on a sampled slice, using a rubric and either human review or a model-as-judge whose own outputs you also log and audit. You do not grade every trace; you grade a stratified sample plus every trace that failed a reference-free check, because those are where the labels are worth paying for. Feed confirmed failures back into the offline set so the next release is tested against them. That loop, not the dashboard, is what keeps the system honest between quarter-ends.

One more thing that is easy to skip and expensive to skip. Tie the trace store to your audit and lineage requirements from day one. When a regulator or an internal reviewer asks why a system produced a particular figure on a particular date, the answer is a trace ID and its full tree, retained under the same controls as the underlying data. If you build observability as a debugging convenience and bolt on retention later, you will discover the useful traces expired the week before you needed them.