Model explainability in finance usually fails the moment it leaves the data science team. The notebook has a tidy SHAP summary plot, the modeller can talk through which features matter, and everyone nods. Then the model goes to validation, or a customer asks why they were declined, and the explanation that felt sufficient turns out to answer a different question than the one being asked.
The problem is rarely a lack of methods. It is that explainability for an audience of one data scientist and explainability for a regulator, a model risk reviewer or a declined applicant are not the same exercise, and people reach for the same tools for both.
Global is not local
Start with a distinction that gets blurred constantly. A global explanation tells you how the model behaves across the population: which features carry weight on average, where the decision boundary sits. A local explanation tells you why this applicant, on this date, got this score. Feature importance and most surrogate models are global. They are useful for understanding the model and for talking to a validator about its overall logic. They are close to useless when you owe one person a reason for one decision.
SHAP sits in an awkward middle. SHAP values are computed per prediction, so they are genuinely local, which is why teams lean on them for reason codes. But they have limits worth stating plainly:
- Correlated features split credit between themselves, so the same economic signal can show up as two half-strength reasons that each look weak.
- The background dataset you choose changes the values. A SHAP explanation is relative to a reference distribution, and people forget to write down which one.
- Approximate SHAP variants can be unstable run to run, which is awkward when the output is a number you publish to a customer.
Surrogate models, fitting an interpretable model to mimic a black box, give you a story that is only as faithful as the surrogate’s fit. Where the surrogate and the real model disagree is exactly where you most need the explanation, and that is where it is least trustworthy. Treat a global surrogate as a sketch of behaviour, not as the reason for any individual decision.
Reason codes are a regulated artefact
For an adverse action notice you do not need a faithful attribution of the model’s internal math. You need a small set of accurate, ranked reasons that a person can act on and that genuinely correspond to why their score fell short. That is a narrower and harder requirement than a SHAP plot. The reasons have to be stable, phrased in factors the applicant recognises, and reconcilable with the model’s actual behaviour rather than a post-hoc rationalisation.
Two explanations that look identical to a modeller can land very differently here. “Utilisation contributed -0.3 to the log-odds” satisfies a data scientist. It does nothing for the customer, and a regulator reviewing a conformity assessment wants to see that the reason codes were generated by a documented, repeatable procedure with an audit trail, not eyeballed from a chart. The gap between those two is where most explainability programmes quietly fall down.
When to skip the explanation entirely
There is a recurring choice that teams defer too long: explain a black box after the fact, or use an interpretable ML model in the first place. If the decision is high-stakes, the population is one you owe reasons to, and the accuracy gap between a monotonic gradient-boosted model or a constrained scorecard and the unconstrained black box is small, the interpretable model is usually the better engineering decision. You get reason codes that are the model, not an approximation of it, and drift is easier to monitor because the structure is legible.
The honest position is that post-hoc model explainability is a tool for cases where an interpretable model genuinely costs you too much performance to accept. It is not a free upgrade you bolt onto any model to make it auditable. Decide which regime you are in before training, not after a validator asks the question you cannot answer.