Batch or real-time? Inference patterns for finance AI

Compute a score in batch when the decision can tolerate data a few hours old. Compute it in real time only when the answer depends on the request itself or on events from the last few seconds. Most finance systems land in between: a heavy base score precomputed overnight, refreshed against a few live features when the request arrives.

The mistake I see most often is treating “real-time” as the default because it sounds modern, then paying for it in latency budget and serving cost for scores that nobody reads within the hour. A borrower’s risk grade does not change between two page loads. Neither does a counterparty’s exposure tier or a vendor’s payment-terms recommendation. What the model can do is beside the point here. The boundary comes from the freshness the decision needs, and if every input to a score was settled before the request arrived, computing it at request time buys you a bigger bill and a harder failure mode.

What the decision can tolerate

The question worth answering is how old the inputs can be at the moment someone acts on the output. Model speed barely enters into it. Write that down per use case before you touch serving architecture.

• A quarterly credit-line review reads a score that can be a day old. Precompute it.
• An AML alert triage queue is populated overnight from settled transactions. Batch is the natural fit, and it lets you run the expensive enrichment once per entity instead of per analyst click.
• A payment authorization decision depends on velocity counts from the last sixty seconds. There is no precompute that saves you here; the signal did not exist when the batch ran.
• A checkout risk check sits in between. The customer’s history is stable and precomputable; the current cart, device and session are not.

Once you frame it as tolerance, the split falls out on its own. The batch side handles everything derived from data that has already reconciled and settled. The real-time side handles only what the request introduces. The boundary between them is where most of the design work goes, because that is where staleness and point-in-time correctness collide.

The other input to the decision is volume shape. If you need scores for every entity in the book on a schedule, batch is the correct pattern regardless of latency, and the lower cost is a bonus. Scoring ten million accounts one HTTP request at a time is an operational choice you will regret at quarter-end. Score them in a single pass, write the results to a store keyed by entity, and let the online path do a lookup instead of an inference.

Precompute, then look up

The batch pattern is simple to state and easy to get wrong. You run the model over a snapshot of features, write each result to a low-latency store with the entity key and an as-of timestamp, and at request time you fetch the stored value. The inference cost moves off the hot path entirely. The online service does a key-value read instead of a model call, so its latency is bounded by the store and its cost per request collapses.

Three things have to be true for this to hold up:

• Every stored score carries the timestamp of the data it was computed from. Without it you cannot answer “as of when was this true,” and you cannot reconstruct a decision for an audit trail or a model-risk review. A stored score with no as-of time will not survive that review; you are asking someone to trust a number with no provenance.
• The batch job is idempotent and reproducible. Rerunning it over the same snapshot produces the same scores. If it does not, you have hidden non-determinism, and your lineage is worthless the first time someone asks why a number changed.
• The features feeding the batch are point-in-time correct. Precompute makes leakage easier to introduce, because it is tempting to join against whatever the warehouse holds now rather than what was known at the as-of cutoff. A batch score built on lookahead data will look excellent offline and fail quietly in production.

Refresh cadence is the tuning knob. Nightly is common because it aligns with settlement and reconciliation cycles. Some scores justify intraday reruns. A few need nothing more than weekly. Pick the cadence from how fast the signal moves, and stay honest that a stored score is a claim about the past, stamped and served.

The hybrid path most systems need

Pure batch is too stale for anything reacting to current behaviour. Pure real-time is too expensive and too fragile for scores that mostly do not change. So the common shape is a base score precomputed in batch, combined at request time with a thin layer of fast-moving features.

Concretely: the overnight job computes the heavy part, the piece that needs the full feature set and the expensive model. That result lands in the online store. When a request arrives, the service reads the base score and blends it with a few live signals, the current transaction amount, a velocity count from a streaming window, a device fingerprint seen thirty seconds ago. The request-time computation stays small, so the latency budget stays intact, and you still react to what just happened.

This is where train/serve skew hides. The feature you compute in a streaming window at request time has to match the feature your batch pipeline reconstructs point-in-time during training. If the streaming counter and the training counter are built by different code, they drift, and your offline metrics start lying to you. Same feature, one definition, exercised by both paths. That discipline is why a feature store earns its keep in this architecture.

One more thing worth stating plainly: the split is a cost lever as much as a latency one. Moving inference to batch means you run the model on your schedule, on hardware you can size for throughput, off the request path where a traffic spike would otherwise multiply your inference bill exactly when you can least afford the load. You give up freshness to get that. Decide the trade per score, deliberately, and stamp what you decided.