Audit trails that make finance AI reproducible

Reproducibility means taking a past decision and rebuilding the exact score the model produced, from the exact inputs it saw, using the model and configuration that were live at that moment. Pull a case from eight months ago, get the same number, then show which features moved it. Most teams discover they cannot do this the first time an examiner asks.

The gap is rarely the model itself. Model weights are usually in a registry somewhere. What goes missing is everything around the prediction: the feature values as they stood at decision time, the threshold that was live that week, the version of the entity-resolution logic that decided two accounts were the same person. Rebuild the decision with today’s versions of those and you get a different answer, and now you are explaining a discrepancy instead of a decision.

What “reproducible” actually requires

A decision is reproducible when you can recover four things and recombine them to the same output. Miss any one and the reconstruction drifts.

• The resolved feature vector. Not the raw source rows, the actual numbers the model consumed after joins, imputation, and encoding. If a feature was computed from a 90-day transaction window, you need that window as it existed at decision time, not as it exists now after backfills and corrections.
• The model version. A content hash of the serialized artifact, not a friendly name like “fraud-v3.” Names get reused. A hash does not.
• The configuration. Thresholds, score cutoffs, the false-positive budget in force, feature flags, any business rules layered on top of the raw score. These change more often than the model and get logged less often than anything.
• The code path. The version of the preprocessing and postprocessing that ran. A change to how you normalize a merchant name can move a match, and that logic usually lives outside the model artifact entirely.

The hard part is point-in-time correctness. Your data changes under you. Corrections arrive late, a counterparty gets re-KYC’d, a transaction gets reversed after the fact. If your audit trail reads features by querying the current state of a table, you have introduced lookahead into your own evidence. The value you show the examiner is not the value the model saw. A feature store with real point-in-time joins solves this for the features it manages, but only if you log the specific snapshot version each decision read, not just “we use a feature store.”

Log the decision, not just the outcome

Most systems log outcomes. The decision was to decline, the score was 0.82, timestamp attached. That satisfies an operational dashboard and fails an audit. To reconstruct why, log a decision record at the moment the decision is made, and make it immutable.

A workable record carries, per decision:

• A stable decision ID that flows through every downstream system, so the alert, the case, the SAR, and the ledger entry all reference the same event.
• The model artifact hash and the config version hash.
• The resolved feature vector, or a reference to an immutable feature snapshot plus the values for anything computed at request time.
• The raw output and the post-processed output, kept separately, so you can see whether the score or the rules on top of it drove the result.
• The code/pipeline version that ran.

Two practices make this hold up. Write the record synchronously, in the same transaction path as the decision, so a decision that ships without a log entry is impossible rather than merely unlikely. And make the store append-only, with hashes, so nobody can quietly edit history after a decision is challenged. A rewritable audit trail carries no weight the moment a decision is contested, because the other side can always argue you edited it after the fact.

For straight-through processing this matters more, not less. When there is no human in the loop, the decision record is the only account of what happened. There is no analyst’s note to fall back on, no memory of the case. If the log is thin, the decision is unexplainable by construction.

Versioning that survives a reconstruction

Versioning is where reproducibility is won or lost, because a decision from last quarter depends on the state of half a dozen things that have all moved since.

Version the model, the training data, the feature definitions, the configuration, and the serving code, and record which version of each was live during any given window. The registry entry for a model should pin the training dataset by hash, so “retrain on fresh data” produces a new lineage node rather than silently overwriting what the old decisions depended on. When someone asks about a decision from a specific date, you resolve the model version live that day, the config live that day, and the feature definitions live that day, then replay.

Replay is the test that tells you whether any of this works. Take a sample of historical decisions, pull their logged inputs and versions, run them back through a reconstructed pipeline, and compare to the stored outputs. If they match to the last digit, your trail is real. If they drift, you have found the seam before an examiner does, and the drift usually points straight at an unversioned dependency: a hardcoded threshold, a library upgrade, a feature that reads current state.

Two things quietly break replay and deserve watching. Non-determinism in the model or the pipeline, where the same inputs give different outputs across runs, which you pin down with fixed seeds and by recording library and runtime versions. And late-arriving data that changes what a point-in-time query returns unless the snapshot it reads is frozen. Both are solvable, but only if you have decided the decision record is a legal artifact and built the pipeline to treat it that way, rather than bolting logging on after the model already ships.