Annual revalidation tells you whether a model was sound last year. It says nothing about whether the model is sound this morning. For a supervised finance model, ongoing monitoring is the part supervisors actually lean on: a running measure of drift and performance, wired to alerts with a named owner, and every check written to an evidence trail an examiner can read without a walkthrough.
Most teams have some version of this. What they usually lack is the join between the three things a reviewer asks for in sequence: did the inputs change, did the model’s behaviour change, and can you prove you would have caught it. Monitoring that answers the first two but leaves the third to a screenshot in someone’s inbox fails the review even when the model is fine.
Watch the inputs before you watch the outputs
Labels are slow. A charge-off, a confirmed fraud case, a matured loan, a SAR disposition. By the time you can compute accuracy, the model has been making decisions on shifted data for weeks. So the first line of monitoring never touches the label. It watches what the model is fed and what it emits.
We instrument three layers:
- Input drift on every feature that carries weight. Population Stability Index and a distributional distance per feature, computed against the training window and against a rolling recent baseline. A PSI over roughly 0.25 on a top-ranked feature is an alert, not a chart nobody opens.
- Score and decision drift. The output distribution, the approval or alert rate, and the reason-code mix. A model can keep a stable AUC while its accept rate quietly climbs three points because one upstream field changed units after a vendor migration.
- Data integrity at the boundary. Null rates, out-of-range values, category cardinality, and stale timestamps. Most “model drift” incidents we are called into are a broken feature pipeline wearing a model’s clothes. A field that started arriving a day late turned a point-in-time feature into a leakage-shaped artefact, and the model’s live behaviour diverged from backtest for a reason that had nothing to do with the model.
The point of the input layer is that it fires before harm is measurable. You get a signal the day the world moves, not the quarter after the losses clear.
Performance monitoring that survives delayed labels
Once labels arrive, you measure the things the validation report measured, on the same eval sets, at a fixed cadence. AUC, KS, calibration, and the metrics tied to the business decision: false-positive rate against the false-positive budget the model was signed off with, and capture rate in the top deciles. The discipline is that the live metric is computed exactly the way the validation metric was. Same segmentation, same exclusion rules, same point-in-time correctness on the features. If the monitoring pipeline reconstructs features differently from the training pipeline, you are comparing two models and calling the gap decay.
For delayed-label models we run a maturing cohort view. Each scoring vintage is tracked as its labels ripen, so a January cohort’s performance is only read once enough of its outcomes are known, and partial reads are marked partial. This stops the two failure modes reviewers push on: declaring victory on immature data, and missing a real decline because the aggregate is diluted by fresh, still-unlabelled cases.
Slice everything. A model can hold flat overall and rot inside one segment that happens to be a protected class or a single high-value counterparty. Population-level metrics hide exactly the failures a fair-lending or concentration reviewer is looking for, so subgroup performance is monitored as a first-class series, not something you compute after a complaint.
The alert has to reach a person with a mandate
A threshold breach that lands in a dashboard nobody is on call for is not monitoring. Each alert is routed to a named owner, carries a severity, and comes with a runbook that says what the first response is: investigate, throttle to shadow, roll back to the prior champion, or escalate to the model risk committee. Champion-challenger structures make the rollback real, because there is always a known-good model to fall back to rather than a scramble.
We tune alerts against a false-positive budget the same way we tune the model. An alerting system that pages the team twice a week for noise gets muted, and a muted alerting system is worse than none because it launders the appearance of oversight. Thresholds get a documented rationale and a review date, so an examiner can see why 0.25 and not 0.2, and who agreed to it.
Evidence is the product
Everything above only counts if you can show it after the fact. The evidence trail is built as the monitoring runs, not assembled the week before an exam:
- Immutable, timestamped records of every metric computed, every threshold in force on that date, and every alert with its disposition and who acted on it.
- Lineage from a live decision back to the exact model version, feature values, and data snapshot that produced it, so a single contested decision can be reconstructed months later.
- A change log tying every threshold change, model promotion, and retraining event to an approval, so the story of the model over time is legible without interviewing the people who were there.
The test we hold this to is plain. An examiner should be able to pick a decision from eight months ago, pull the model version and inputs behind it, see what the monitoring said that week, and confirm that the alerts which should have fired did fire and were handled. If reconstructing that costs you three days of engineering, the trail is not real yet. When the answer is one query, the model is governed whether or not anyone is looking.
FAQ
How often should a supervised finance model be monitored?
Match the cadence to how fast the inputs move. Transaction-monitoring and fraud models need daily or intraday checks; a credit model rescored quarterly can run weekly performance panels. Annual revalidation sits on top of continuous monitoring, it does not replace it.
What is the difference between data drift and performance decay?
Data drift is a change in the input distribution, which you can measure the day it happens. Performance decay is a change in accuracy, which you often cannot measure until labels arrive weeks later. Drift is the early warning; decay is the confirmed harm.
How do you monitor a model when the ground-truth labels are delayed?
You watch leading indicators that need no labels: input drift, score distribution shifts, approval and alert rates, and reason-code mix. Then you reconcile against outcomes once labels land, so the early signals get calibrated against what actually happened.