Scoring a payment for fraud in under a second is mostly a data-movement problem. By the time the authorization message reaches your service you have a hard ceiling, often 200 to 400 milliseconds before the network times out. Inside that window you fetch features, compute a score, apply policy and answer approve or decline. The model is the fast part.
Everything that feeds the model is where the engineering goes. Most teams underestimate this because the model is the visible artefact. It has an AUC, a training notebook, a story to tell in review. The feature path has none of that, yet it is where authorization fraud systems live or die. So the architecture below spends most of its attention there.
The latency budget decides the design
Start by writing the budget down as line items, because the total is what the card scheme or payment rail gives you, and it does not care about your excuses. A realistic split for a 300ms end-to-end ceiling might look like this:
- Network and deserialization in and out: 30-50ms you do not control much.
- Feature retrieval from the online store: 40-120ms, the biggest variable.
- On-the-fly feature computation and enrichment: 20-80ms.
- Model inference: 2-15ms for a gradient-boosted tree, more if you insist on a deep model.
- Policy, thresholds and response assembly: 10-30ms.
The moment you see these numbers, two decisions follow. First, the time left after feature retrieval sets your model budget, so the leaderboard score is beside the point. A boosted tree that scores in 5ms and explains itself beats a neural model that needs 60ms and a GPU you have to keep warm. Second, feature retrieval has to be a single batched lookup against a low-latency key-value store rather than a series of round trips. If your features live in six systems, you have already lost.
The tail is what matters, not the average. A p50 of 80ms with a p99 of 900ms means one in a hundred payments breaches the ceiling, and those breaches cluster exactly when volume spikes, which is when fraud pressure is highest too. Budget against p99.
The streaming feature path
The features that actually catch authorization fraud are behavioural and recent. Velocity counts (how many transactions on this card, this device, this merchant in the last 60 seconds, 10 minutes, 24 hours), amount deviation from the entity’s own history, first-seen relationships between card and merchant, geographic impossibility. All of these are aggregations over a stream of events, and they need to be current to the last few seconds. A batch feature computed overnight is useless against a card that started being tested twenty minutes ago.
So you run two paths that must agree. A streaming job consumes the authorization event stream and maintains windowed aggregates in the online store. A training pipeline reconstructs those same aggregates point-in-time over historical events. The trap is building these with different code. When the streaming counter and the offline counter diverge, you get train/serve skew, and your model has learned from features it will never actually see at decision time. The defence is a single feature definition that both paths execute, and a reconciliation job that samples live-scored transactions, recomputes their features from the log, and alarms when the two disagree beyond a tolerance.
A few things we insist on in this layer:
- Entity resolution happens before aggregation, not after. If the same card or device appears under two identities, your velocity counts undercount, and undercounting velocity is exactly how test-card attacks slip through.
- Every feature carries a freshness timestamp. A stale feature is worse than a missing one, because the model trusts it. The scorer should know the age of what it is reading.
- Feature computation is versioned and lineage-tracked. When a fraud analyst asks why a payment was declined, you need to reproduce the exact feature vector as of that millisecond, or the audit trail is fiction.
Design the degraded path before you need it
The online store will time out. A streaming job will lag during a deploy. A downstream enrichment call will hang. In a batch system you retry; at authorization time you have no such luxury, so degradation has to be a first-class part of the design.
We build the decision engine to accept a score plus a confidence context, and to behave differently when the context is degraded. If velocity features are missing because the store timed out, the engine leans harder on the static rule layer and the features that did arrive, and it shifts the threshold to be more cautious rather than defaulting to approve. Failing open floods you with fraud; failing closed blocks good customers and burns your false-positive budget. The right posture depends on the merchant, the amount and the risk appetite, so it belongs in policy, not buried in a catch block.
The false-positive budget deserves explicit accounting. Every decline of a legitimate payment has a cost, and unlike fraud losses it is mostly invisible on the dashboard. Set a target block rate for good traffic, monitor it per segment, and treat a breach as an incident. A model that quietly starts declining 3% of a good merchant’s traffic instead of 0.5% is a production failure even if fraud losses look fine.
Finally, watch for drift on the inputs, not just the outputs. Fraud is adversarial; the distribution moves because someone is actively probing it. Monitor feature distributions and score distributions in near-real-time, and keep a labelled eval set that you refresh as chargebacks and confirmed-fraud labels arrive weeks later. Those late labels are the only ground truth you get, and the lag between a decision and its label is the hardest constraint in the whole system to work around.
FAQ
How much of the latency budget should the model actually get?
Usually the smallest slice. On a 300ms budget we often see feature retrieval and enrichment eat 150ms or more, so a gradient-boosted model that scores in single-digit milliseconds is rarely the bottleneck. Spend your optimisation effort on the feature path first.
What happens when the feature store times out mid-authorization?
You cannot block the payment network waiting. The system falls back to a degraded score computed from whatever features arrived in time, plus a static rule layer, and the decision engine treats a fallback score more conservatively than a full one. The event is flagged for review and logged so you can measure how often it happens.
Do streaming features cause train/serve skew?
They can, and it is the most common way these systems go wrong. The counter you compute in a streaming window at authorization time must match the counter your training pipeline reconstructs point-in-time. If the two are built by different code, they will drift apart and your offline metrics will lie.