Data quality observability for finance AI

Data quality observability for finance AI is the standing telemetry you put on the data feeding a model so a broken input is caught at the source, not inferred from a wrong answer downstream. In practice that means monitoring four things on every table the model reads: freshness, volume, schema, and the distribution of the values themselves. The point is to fail loudly at ingestion instead of quietly at inference.

The failure mode that gets people is not the pipeline that crashes. A crashed job pages someone. The dangerous one is the pipeline that keeps running against a feed that stopped updating, or silently swapped a currency, or started sending null where it used to send zero. Nothing throws. The tables populate. The model scores. Three layers downstream, a risk figure is off by a factor you cannot explain in the quarter-end review, and now you are reading commit logs backwards trying to find where the number went wrong.

Freshness is the first thing that breaks and the last thing people check

Most finance data has a heartbeat. A vendor price file lands at a known time. A ledger extract runs on a schedule. A KYC provider pushes updates on their cadence, not yours. When that heartbeat skips, the data does not disappear; it goes stale, which is worse, because stale data looks exactly like fresh data to everything downstream.

We set a freshness budget per source, and we set it against the source’s actual behaviour rather than a round number someone liked. If a vendor publishes end-of-day marks at 18:05 New York time on business days, the check knows about the time zone, knows about the holiday calendar, and knows not to expect a file on a market holiday. The alert fires when the newest partition is older than the budget allows, and the downstream job that consumes it blocks on that check instead of proceeding.

• Key the SLA to the real publish cadence, not to when your job happens to run.
• Carry the source time zone through the check. “Yesterday” is a different instant in three data centres.
• Distinguish “late” from “missing.” A file that is twenty minutes late is a warning. A file that never came before the model’s cutoff is a hard stop.
• Watch point-in-time correctness here too. A feed that backfills historical rows after the fact will quietly rewrite what the model would have seen on a past date, which is how leakage sneaks into a backtest.

Volume and schema catch the failures that still pass every type check

A row count is a blunt instrument and it is one of the most reliable signals you have. When a table that lands two million rows a day lands sixty thousand, something upstream truncated, and no per-row validation will tell you, because each of those sixty thousand rows is individually valid. So we track expected volume as a range that accounts for the weekly and monthly shape of the business. Settlement volume is not flat across the week. Reporting spikes at quarter-end. A static threshold either screams every Monday or misses a real drop, so the band has to know the calendar.

Schema is the other silent killer. A vendor renames a column, changes a decimal’s scale, or promotes an enum to a free-text field, and if you are reading by position or coercing types loosely, the pipeline absorbs it and moves on. We pin schemas explicitly and treat any drift as an event that needs a human decision, because a schema change is often a data contract change that nobody told you about.

• Type and nullability per column, checked on arrival, not assumed.
• Cardinality on categorical fields. A currency column that suddenly holds forty distinct values instead of twelve is a merge gone wrong or an entity resolution failure upstream.
• Units and scale. Basis points versus percent, cents versus dollars. These pass every type check and destroy every calculation.
• Primary-key uniqueness and referential integrity, so a duplicated join key does not double a position.

Distribution monitoring is where you find the problems nobody wrote a rule for

Freshness, volume, and schema catch structural breaks. They do not catch the feed that is on time, full, correctly shaped, and wrong. For that you have to watch the statistical shape of the values and alert when it moves.

This is a close cousin of feature drift monitoring, but it sits earlier, on the raw inputs rather than the engineered features. We baseline each numeric column’s distribution and each categorical column’s frequencies over a trailing window, then compare incoming data against that baseline. When a transaction-amount distribution shifts its mean, or a country-code field’s mix changes overnight, the detector flags it for review before it propagates into a feature store and from there into every model that reads those features.

The hard part is not the detection. It is the false-positive budget. Finance data is genuinely non-stationary: rates move, volumes seasonally swing, a real market event will look exactly like an anomaly for a day. An observability layer that pages on every legitimate shift gets muted within a week, and a muted alert is the same as no alert. So the thresholds are tuned to the series, seasonality is modelled rather than fought, and known events are annotated so the system does not relearn them as anomalies every time.

Two practices keep the signal honest. First, every alert carries lineage, so the on-call engineer sees which upstream source and which transformation produced the anomalous table and does not spend twenty minutes finding it. Second, every incident gets written back as a labelled example, and that set of confirmed real breaks versus benign shifts becomes the eval set you tune the detectors against. Without it you are guessing at thresholds and calling the guess a policy.

None of this replaces the reconciliation and controls a finance function already runs. It sits in front of them, so that by the time a number reaches a model, a report, or a straight-through decision, the data behind it has already survived the checks that would otherwise only run after someone noticed the answer looked wrong. The audit trail then shows not just what the model decided, but that the inputs it decided on were fresh, complete, correctly shaped, and inside their expected distribution at the moment of the decision.