A data contract is a versioned, enforced agreement about a feed: its schema, the meaning of each field, freshness and null expectations, allowed value ranges, and its owner. In a finance AI pipeline it sits where the feed enters your control and fails loudly when reality drifts from the agreement, so a change three teams upstream cannot quietly poison a production model.
The failure it prevents is specific and common. A vendor renames a column, changes a currency code from ISO alpha to numeric, or starts sending prices as strings with thousands separators. Nothing throws. The load job succeeds, the feature job succeeds, the model scores. Two weeks later someone notices the false-positive rate on a screening model climbed, or an exposure figure in a report is off by a factor no one can explain. By then the bad values sit in your feature store, your eval sets, and probably a few decisions you cannot unwind. The expensive part is rarely the downtime. It is the days spent proving which of forty upstream fields moved, and when.
What a contract actually contains
A schema tells you a field is called settlement_amount and is a decimal. That is the easy part and it is not where finance data hurts you. The damage lives in semantics and timing, so the contract has to cover more than shape.
- Meaning, not just type.
priceis meaningless without knowing it is clean or dirty, mid or last-trade, in which currency, and per what quantity. We write these down as field-level assertions and reject rows that violate them. - Freshness and completeness. A feed that is normally 30 seconds behind and suddenly runs six hours stale is broken even though every value is well-formed. The contract states the expected lag and the expected row count per window, and a shortfall fails the check.
- Value-range and referential rules. Currency codes must be in a known set. Instrument identifiers must resolve against the security master. A negative quantity on a feed that never carries shorts is a defect, not a data point.
- Point-in-time semantics. For any field that gets restated, the contract records whether a value is final or provisional and forbids a silent overwrite of history. This is the part that protects you from lookahead: if a restated figure can quietly replace what was true as-of the decision date, your backtest is fiction.
- An owner and a version. Every contract names a human or team accountable for changes, and a version number. Breaking changes require a version bump, which is the hook the rest of the system watches.
Enforce at the boundary, and make breakage loud
The rule we hold to: validate a feed the moment it enters your control, before a single feature reads it. Push enforcement downstream and you are inspecting the patient after the infection has spread. Catch it at ingestion and the blast radius is one feed on one day.
Concretely, that means a validation stage between raw landing and anything your models touch. It checks incoming batches or messages against the contract and routes failures to a quarantine rather than the main table. A quarantined batch does not silently disappear and it does not silently pass. It halts the affected partition and pages the owner with the exact assertion that failed and three sample rows that broke it. The point is to make a contract violation as visible as a failed unit test, because that is what it is.
Two design choices matter here. First, distinguish a hard stop from a soft flag. A currency code you have never seen is a hard stop; you do not want to guess. A freshness lag slightly over budget might be a warning that still lets scoring proceed on a degraded-mode flag. Encode that policy per assertion rather than treating every deviation as an outage. Second, keep the check cheap enough to run on every batch. If validation is expensive, it gets sampled, and a sampled check misses exactly the intermittent corruption that is hardest to trace later.
Contracts also give you a real lineage story. When a downstream number looks wrong, you can point at the boundary where a specific field last passed its contract and where it first failed, with a timestamp. That collapses the investigation from “search the whole pipeline” to “read the quarantine log.” For anything that has to stand up to an audit trail, the same records show a reviewer that the feed feeding a decision met its stated expectations on the date the decision was made.
Versioning is the hard part, not validation
Writing assertions is straightforward. The engineering that earns its keep is handling the day the upstream owner has a legitimate reason to change the feed. They will. A vendor adds a field, splits one into two, or changes a code list. A contract that cannot absorb this becomes something people route around, and a bypassed contract is worse than none because it lends false confidence.
So we treat a contract as a versioned interface with a deprecation path. An additive change is backward-compatible and bumps a minor version; consumers ignore the new field until they choose to read it. A breaking change bumps a major version and both versions run in parallel for an agreed window while consumers migrate. The producer cannot unilaterally break the old version. That negotiation is the whole point of calling it a contract rather than a validation script.
A few habits keep this from rotting:
- Test the contract against real history. Replay a month of past feeds through a new contract version before it goes live. If yesterday’s legitimate data fails tomorrow’s rules, the rules are wrong, and you find out before production does.
- Version the contract with the data, not separately. The version that validated a batch should be recorded alongside that batch, so a year later you can reconstruct exactly which rules were in force.
- Keep a tight false-positive budget on the checks themselves. A contract that cries wolf on normal quarter-end volume spikes gets muted, and a muted contract enforces nothing. Tune ranges against seasonal reality, including the corporate-actions and month-end patterns that make finance feeds lumpy.
None of this makes the upstream world stable. It makes upstream instability something your pipeline observes and reacts to on a schedule you control, instead of something a model discovers for you after the fact.
FAQ
Is a data contract the same as a schema?
No. A schema describes shape. A contract adds semantics, freshness expectations, allowed value ranges, an owner, and a versioning policy, and it fails the build when any of those are violated.
Where do you enforce a contract in the pipeline?
At the boundary where a feed enters your control, before any feature or model reads it. Enforcing later means the bad data has already spread and you are debugging symptoms instead of the cause.
Do contracts help with point-in-time correctness?
They help you catch the setup for leakage, such as a restated value overwriting history or a field arriving late. They do not replace an as-of store, but they make the store's assumptions checkable.