Monitoring feature drift before it breaks the model

Feature drift is a change in the distribution of a model’s inputs after it reaches production. The model keeps returning confident scores even when the data underneath it has moved somewhere it was never trained on. Catch it early by monitoring each feature’s distribution against a fixed baseline at the point the feature is served, so you alert on the input before a wrong output reaches a decision.

Most model incidents in finance do not announce themselves. The API stays up, latency is fine, the score comes back between zero and one, and nothing in the dashboard turns red. The thing that changed is upstream: a vendor started sending a field in cents instead of dollars, a merchant category code got remapped, a KYC provider swapped its risk buckets, an ingestion job started dropping a segment of customers on the floor. The model has no way to tell you it is now extrapolating. It just answers. By the time the label comes back and the AUC sags, you have made three weeks of decisions on inputs the model did not understand.

What actually moves, and why label-level metrics find it late

The reason to watch features rather than only outputs is timing. In a lending or fraud setting the ground-truth label arrives long after the decision. A charge-off is known months later; a disputed transaction resolves in weeks; a manual review closes when someone gets to it. If your only monitor is model performance against labels, you are looking at a lagging indicator of a problem that started at ingestion.

Feature drift, by contrast, is observable the instant the feature is computed. You do not need the outcome. You compare what you are serving today against what the model saw in training and in the stable weeks after launch. The kinds of movement worth separating:

• Covariate shift: the input distribution moves while the true input-to-outcome relationship holds. Incomes climb over a year, or a new acquisition channel skews the population younger than the one you trained on.
• Schema and semantic breaks: a unit changes, an enum gains a value, a nullable field starts arriving null far more often. These are not really drift, they are pipeline defects wearing drift’s clothes, and they are the most common cause of a model quietly going wrong.
• Upstream feature-logic changes: someone edits an aggregation window or a join key three systems away, and a feature that used to mean thirty-day spend now means ninety. Without lineage you will read this as the world changing rather than your own code changing.

Separating these matters because the fix differs. Covariate shift may call for a retrain. A semantic break calls for a rollback and a conversation with the data owner. Treating the second as the first is how teams retrain models on corrupted inputs and bake the defect in.

How we instrument it

The unit of monitoring is the feature as it is served, not the raw source table. We tap the values at the point the model reads them, which for a serving path usually means logging feature vectors at inference and, where a feature store exists, reading the same materialized values the model consumed. That last point is the whole game: if your monitor computes drift from a separate query against production tables, you are measuring a different number than the model saw, and point-in-time correctness quietly slips. The monitor has to observe the exact bytes that hit the model.

From there:

• Baseline windows. Freeze a reference distribution per feature from the training set and, separately, from a stable post-launch window. Two references, because the training set can already differ from live traffic on day one, and you want to know which surprise you are looking at.
• Numeric features. Track the distribution, not just the mean. A feature can hold its average while its tails move, and the tails are where credit and fraud decisions live. We use population stability index and a distance on the empirical distribution, binned once against the baseline so the bins do not drift with the data.
• Categorical features. Watch the category mix and, specifically, the arrival of unseen categories. A new merchant code the model has never encountered is a stronger signal than a two-point wobble in an existing one.
• Missingness and volume. Null rate and row counts per segment, monitored on their own. A feature does not have to change value to break a model; it only has to stop arriving for one population.

Thresholds get set against a false-positive budget, not a textbook constant. Pick a number of alerts per week the on-call can genuinely investigate, then fit the bands to hit roughly that under normal conditions. Seasonality is real in this data. Quarter-end moves balances, payroll cycles move transaction volume, tax season moves everything. Bands that ignore the calendar will page you for the calendar.

Wiring drift into a response, not just a chart

A drift number that no one is obligated to act on is decoration. The monitor earns its place when it is connected to something that happens.

• Every alert carries lineage: which feature, which upstream job produces it, which model versions consume it, when it last changed. Whoever gets paged should not have to reverse-engineer the dependency graph at 2am.
• Alerts route by feature ownership, not to a generic channel. The team that owns the KYC feed hears about the KYC feature. That is also the audit trail a model-risk reviewer will ask for later.
• Sustained drift on a feature the model leans on hard should be able to pull a lever automatically: widen a review queue, lower a straight-through-processing threshold, or shift traffic to a challenger running in shadow. The safe default when inputs leave the known region is to send more decisions to a human, not fewer.
• Drift events get logged with the same permanence as decisions. When a regulator or an internal review asks why a batch of approvals looked odd in June, “feature X drifted on the 9th, we widened manual review on the 10th, retrained on the 24th” is the answer you want to have written down at the time, not reconstructed after.

The goal here is narrow, and worth stating plainly. No monitor keeps a model right when the world turns against it. What it buys you is warning. When the ground moves, you learn it from the inputs while there is still time to act, rather than from a delinquency report a quarter later.