Building point-in-time feature pipelines

Point-in-time feature pipelines answer one question for every value they produce: was this known at the moment we are trying to predict? A feature is point-in-time correct when it only ever reflects data that had actually arrived and been recorded as of the event timestamp. Get that constraint wrong and your model learns the future, scores beautifully offline, and falls apart the day it goes live.

The failure is quiet because nothing errors. The join runs, the numbers look plausible, the notebook cell turns green. The problem is that the number is one a human sitting at the decision point could never have seen. In finance this is not an academic nicety. It is the difference between a fraud model that flags a transaction before authorization and one that quietly trained on the chargeback that arrived six weeks later.

Where the future leaks in

Leakage is rarely a dramatic bug. It arrives through ordinary joins against tables that only hold the current state of the world. A customer’s risk tier, an account’s balance, a merchant category, a KYC status: all of these get updated in place. When you join them to a historical event by key alone, the pipeline hands the model today’s value for a decision that happened last March.

The patterns we see most often:

• Target-derived features. A field is populated when the case resolves, not when the decision was made. chargeback_amount, days_to_default, final_disposition. These correlate perfectly with the label because they partly are the label.
• Aggregates over the wrong window. A 30-day transaction count computed with a window that runs from the event forward, or one that silently includes the event being scored.
• Restated source data. A vendor sends a figure, then corrects it a week later. If you store only the latest version, every backtest reads the correction rather than the original print. Quarter-end restatements are a common offender.
• Slowly changing dimensions treated as static. An address, an industry code, a beneficial-owner record that changed after the event but overwrites the row you join to.

None of these throw. All of them inflate offline metrics and none of the lift survives contact with production.

The knowledge timestamp is the load-bearing column

The fix starts with admitting that most finance data has two clocks. There is when something happened, the event or effective time. And there is when your systems found out, the knowledge or ingestion time. A wire settles on Monday but posts to your ledger Tuesday afternoon. A bureau refreshes a score with a reference date of the first but you receive the file on the fourth.

A point-in-time correct feature must filter on the second clock. When you build a feature for an event at time t, you may only use records whose knowledge time is at or before t. Event time alone is not enough, because late-arriving and restated records carry an event time in the past while only becoming known to you later.

Concretely, we require every feature source to carry an explicit valid_from and an ingestion or knowledge timestamp, and we treat any table without one as unsafe to join. Reference and dimension data gets stored so that prior versions survive a restatement rather than being overwritten. Where a source restates history, that means bitemporal storage: we can reconstruct both what was true and what we believed on any past date. It is more work at ingest, and it is the only way a backtest tells the truth.

As-of joins, done so they hold at scale

The mechanical core of the pipeline is the as-of join. For each labeled event, you reach into a feature source and pull the last value whose timestamp is less than or equal to the event’s cutoff. Not equal, not nearest, and never after. Done row by row this is trivial and unbearably slow; done as a set operation it is a sorted merge on the time key within each entity.

A few rules we hold to:

• One cutoff per event, computed once. Every feature for a given event resolves against the same as-of timestamp. If different features use different cutoffs, you cannot reason about what the model saw.
• The same code path offline and online. The transform that builds training rows and the one that serves a live request should be the same function reading the same definitions. When they diverge, training-serving skew creeps in and it looks exactly like leakage.
• A deliberate feature delay. If a signal typically lands two days after its event time, we sometimes hold features back by that lag even in training, so the model learns on the freshness it will actually have at inference. Optimism about data arrival is its own form of leakage.
• Lineage you can point an auditor at. Each feature value should trace to the source records and versions that produced it, at the timestamps that were in force. Under SR 11-7 model risk expectations, and for anything that touches an adverse decision, that trail is not optional.

Proving the pipeline is honest

You cannot eyeball point-in-time correctness, so we test for it. The most useful check is a negative control: build a feature deliberately from post-event data, confirm it lights up the model, then confirm the audited pipeline refuses to produce it. If a known-leaky feature and your production feature score the same, your production feature is leaky.

Beyond that, we hold out an eval set strictly forward in time from the training window, never a random split, so the evaluation mirrors the real deployment order. We reconcile feature distributions between the training build and the online store and alarm on drift between them, because a gap there usually means the two code paths have parted ways. And we keep a small panel of hand-traced events where an analyst has written down, by hand, what was knowable at the decision point, and we diff the pipeline against that ground truth whenever the definitions change.

None of this is glamorous. It comes down to timestamps, versioned dimensions, and joins that respect a cutoff. But it decides whether the model is honest. A model is only ever as honest as the moment its features were allowed to see. When a finance model sees too much, the cost lands as a real decision on information that did not yet exist: a loan approved or a payment cleared on a fact nobody at the desk could have had.