Running AI incident postmortems in finance

A model incident without a postmortem is one you will repeat, because the conditions that produced it still sit in your pipeline. The postmortem reconstructs what the model did, why the surrounding controls let the output through, and which change stops the same failure mode next quarter. Run it blameless, and tie every action item to an owner and a date.

The reflex in finance is to treat a model error as a validation problem: retrain, revalidate, close the ticket. That misses most of what went wrong. A wrong score rarely reaches a customer on its own. It reaches them because a monitoring threshold was set too loose, a fallback did not trigger, a reconciliation ran after the payment instead of before it, or nobody owned the alert that did fire. The model is one link. The postmortem is where you examine the whole chain. Run one on the near misses too. A bad score that a reconciliation happened to catch is telling you the control worked by accident, and accidents do not repeat on schedule.

Reconstruct the timeline before you argue about cause

The first job is a factual timeline that everyone in the room agrees on, built from the audit trail rather than from memory. This is where reproducibility earns its keep. If you cannot replay the exact inputs the model saw at the moment it produced the bad output, you are guessing, and a room full of senior people guessing tends to converge on whoever spoke last.

Pull the record together first:

• The exact feature vector served, with the feature store version and the point-in-time values, not the values as they read today after backfills and corrections.
• Model version, weights hash and the config that was live, so you can rule out a silent deploy or a rollback that half-completed.
• Every downstream control the output passed through: score thresholds, human review queues, reconciliation jobs, straight-through processing rules.
• The wall-clock sequence. When the bad output was produced, when it acted on something, when a human or a monitor first noticed, and how long it ran unnoticed.

That last gap, detection latency, is usually the most damaging number in the whole write-up and the one people most want to skip past. A model that is wrong for four hours during a quarter-end batch is a very different incident from one that is wrong for four weeks across every scored application. Write the timeline against the audit trail as you go, with timestamps and log references next to each step, so the reconstruction survives the meeting and stands up if a validator or a supervisor asks to see it later.

Keep it blameless, and mean it

A blameless review is the only way to get a true timeline out of the people who were there. The moment an engineer believes the review exists to assign fault, they narrate defensively, and you lose the detail that tells you how the failure actually propagated. You end up with a document that protects careers and teaches nothing.

The discipline is to phrase every finding as a property of the system, not the person. “The on-call had no runbook for a drift alert on this model” is useful. “The on-call ignored the alert” is a dead end, because the next on-call is a different person facing the same missing runbook. When someone made a call that looks wrong in hindsight, the question is what information they had at the time and why the environment made that the reasonable choice. Almost always the answer points at a gap you can close with code or process, which is the whole point.

Blameless does not dissolve accountability for the model itself. Under SR 11-7 and DORA the named owner still owns it. You are separating two things that finance tends to fuse: who is answerable for the control, and how the failure mechanically happened. Fuse them and the second question never gets an honest answer.

Find the mechanism, not the trigger

Root cause in a model incident is rarely a single line. Ask why until you reach something you can actually change, and stop when the next why leaves your system boundary.

A worked example. A transaction-monitoring model started clearing payments it should have flagged. The trigger was an upstream schema change that renamed a counterparty field. But the trigger is not the mechanism. The field went null, the feature silently imputed to a default, the default happened to look low-risk, and the drift monitor watched score distributions rather than input completeness, so nothing fired. Four causes, four different fixes, and only one of them is “coordinate schema changes.” The others are a null-rate check on ingestion, an imputation policy that fails loud instead of guessing, and a monitor that watches input health and not just outputs.

Good postmortems produce a small number of specific changes, each with an owner and a date, each verifiable:

• A control that would have caught this exact failure, expressed as a check you can test, not an intention to be careful.
• An eval case added to the regression set that encodes the failure, so a future model has to pass the scenario that broke this one.
• A monitoring change with a concrete threshold and a named alert owner, closing the detection-latency gap the timeline exposed.

Judge a postmortem by one question: would the action items, once shipped, have stopped this incident? Explaining what happened is the easy part. If you cannot point to the specific check that now fires where nothing fired before, you have written a description, not a fix, and the same incident is still in front of you.