Reference data management for finance AI means keeping the slow-moving facts a model treats as ground truth versioned and point-in-time-correct: currency codes, calendars, instrument identifiers, classifications, counterparty hierarchies. These rarely change and are almost never what you are modelling, which is exactly why they break things quietly. A wrong calendar throws no error. It just makes every downstream number invisibly wrong.
The full list is longer than most teams expect: currency codes, market and settlement calendars, instrument identifiers, sector and country classifications, counterparty hierarchies, product and account taxonomies. Nobody ships a pipeline that reads USD as EUR. What actually happens is subtler. An exchange renames a ticker, a country splits an ISIN range, a holiday gets added to a settlement calendar mid-year, and three of your feeds pick up the change on three different dates. Your model keeps running. Its inputs have shifted underneath it, and the first sign is a slow drift in predictions that someone eventually blames on the model instead of the data feeding it.
Point-in-time correctness is the whole game
The single most expensive mistake in finance ML is training on reference data as it looks today rather than as it looked on the historical date you are pretending to be. A stock’s current sector classification, its current identifier, its current parent company: none of that was necessarily true two years ago. Join today’s reference table onto a two-year-old feature and you have leaked the future into the past. The backtest looks great. Production does not match it, and now you are debugging a model when the bug is in a join.
We treat reference data as bitemporal by default. Every record carries two clocks:
- Effective time: when the fact became true in the world. A ticker rename is effective on the exchange’s change date.
- Knowledge time: when we learned the fact. A vendor may deliver that rename three days late, or backdate a correction.
With both clocks you can answer the two questions that matter. What was true on date X, and what did we believe was true on date X? Training and backtests use the second one, reconstructing exactly the reference state the model would have seen, late corrections and all. Serving uses the current best knowledge. Keeping those separate is what stops lookahead from creeping in through the reference layer, which is the one place teams almost never think to check.
Practically this means no destructive updates. Reference tables are append-only histories with valid-from and valid-to ranges. A correction closes the old row and opens a new one; it never overwrites. Storage is cheap and the audit trail pays for itself the first time someone asks why a number changed.
Managing the mappings, not just the codes
The codes themselves are the easy part. ISO 4217 for currency, ISO 3166 for country, ISO 10383 (MIC) for venues, LEI for legal entities, ISIN and FIGI for instruments. The hard part is the mappings between them, and the fact that the same real-world thing wears a different name in every feed.
One instrument might arrive as an ISIN from a custodian, a vendor-proprietary ID from a market data feed, an internal security ID from the book of record, and a free-text description from a spreadsheet a desk still maintains by hand. Getting these to agree is entity resolution, and it is where most reference data projects actually spend their time. A few rules we hold to:
- One golden key per concept, with every source identifier mapped to it and the mapping itself versioned. Downstream systems reference the golden key, never a vendor’s ID directly.
- Cross-references are many-to-many over time. An ISIN can point to different instruments across a reuse gap; a FIGI is more stable but not universal. Model the relationship as time-bounded edges, not a static lookup.
- Unmatched records go to a review queue, not to a silent default. A default currency or a fallback calendar is how wrong numbers enter the system without anyone deciding to let them in.
Calendars deserve their own mention because they are deceptively fiddly. Trading calendars, settlement calendars, holiday calendars and business-day conventions differ per market and change over time. Get a single holiday wrong and you shift every lagged feature, every rolling window, and every settlement date you compute for that market, whether it settles T+1 or T+2. We source calendars per venue, version them, and test date arithmetic against known settlement dates rather than trusting a library’s defaults.
Governance that survives quarter-end
Reference data goes wrong at the boundaries. A vendor changes a schema, a new market gets onboarded, a corporate action reshapes an identifier, and it always seems to land during quarter-end when nobody has time to notice. The controls that catch this are unglamorous, and they are the ones we insist on.
- Reconciliation between sources on a schedule. When the custodian’s currency for a position disagrees with the market data feed’s, that is a finding, logged with a false-positive budget so real breaks do not drown in noise.
- Lineage from every model input back to the source record and the exact version used. When a prediction looks off, the question “where did this code come from and when did it change?” should take minutes, not a day.
- Change alerting on the reference layer itself. A spike in new identifiers, a calendar edit, a mapping that suddenly covers fewer records than yesterday: these are early warnings that something upstream moved.
- Ownership. Someone signs off on new code sets and mapping changes. Reference data with no owner rots, and the rot is invisible until a model eats it.
None of this is exciting work. It is plumbing. But the model is only as point-in-time-correct as the reference data underneath it, and that data will drift the moment you stop watching it. The teams that get consistent, reproducible model behaviour are usually not the ones with the cleverest architecture. They are the ones who treated currencies, calendars, instruments and codes as a versioned and reconciled asset with a clear owner, instead of a lookup table someone set up once and forgot.
FAQ
How do you version reference data so a model can be reproduced later?
Store every code set, mapping and calendar as an append-only history with valid-from and valid-to timestamps, plus the ingest time. To reproduce a run you query the data as it stood at the run's decision time, not as it stands today.
Do we need a bitemporal store, or is a plain effective-date column enough?
If you only ever answer questions as of now, an effective-date column is enough. The moment you need to explain what a model saw on a past date, or defend it in an audit, you need both effective time and knowledge time, which is bitemporal.
Where should reference data live relative to the feature store?
Upstream of it. Resolve identifiers, apply the correct calendar and normalise codes before features are computed, so the feature store only ever sees clean, point-in-time-correct keys. Fixing reference data inside feature logic scatters the same correction across every pipeline.