Champion/challenger and shadow deployment for underwriting models

You validate a new underwriting model in production by running it alongside the live one where it decides nothing. The champion keeps making the real calls. The challenger scores the same applications in the background, and its decisions land in a log rather than on a customer. You compare the two on live traffic, wait for outcomes to season, then route real decisions across. That is shadow deployment.

Why go to this trouble? Because a lending decision is not reversible the way a checkout-button color is. Decline someone who should have been approved and you lose the customer and possibly draw an adverse action complaint. Approve someone the challenger would have declined and you carry the loss on the book for years. A careless A/B test on a live population converts model uncertainty straight into credit losses and conduct risk. Shadow mode buys you evidence before you spend anyone’s money.

Shadow first, then a metered challenger

Shadow deployment means the challenger consumes the exact feature vector the champion saw, at the same instant, and produces a score and a decision that go nowhere except a log table. This is the safest stage. It is also the one people get wrong most often.

The trap is point-in-time correctness. Your challenger usually needs features the champion never used, and it is tempting to backfill them from the current state of your warehouse. Don’t. If a feature reads a value that was only known after the application was decided, you have leakage, and the shadow results will look wonderful for reasons that will never reproduce in production. The discipline is the one you already owe your training set: every feature the challenger reads has to be materialized as of decision time, ideally from the same feature store the champion pulls from, with the same freshness and the same null-handling.

Once the shadow numbers hold up, you move to a live challenger, but a metered one. Route a small, randomly assigned slice of traffic to let the challenger actually decide. Keep the slice small, and keep the assignment stable per applicant so a customer who reapplies does not bounce between models. Ring-fence the exposure:

• Cap the challenger’s share of new bookings and the dollar exposure it can originate.
• Exclude segments where a wrong decision is expensive or sensitive, at least at first.
• Set kill-switch thresholds on approval rate, average score, and override rate that halt routing automatically.

What you actually compare

The honest comparison is not “which model has the higher AUC on last quarter’s data.” That was settled before you deployed anything. In production you are testing whether the offline story survives contact with live traffic, and whether the two models disagree in ways that matter.

Score agreement is the first cut. Compute the population stability index between champion and challenger score distributions, then read the disagreement matrix: the cells where one approves and the other declines. Disagreement concentrated in a single segment is a signal, usually a feature that behaves differently in production than it did in your sample.

The decision that counts is on matured outcomes. A challenger looks great on day one because you cannot yet see who defaults. The outcomes have to season. That means holding judgment until enough of the shadow-scored accounts have had time to go bad, which on a consumer installment book runs to months. Until then you are comparing proxies, and you should say so out loud. What is worth tracking through the wait:

• Approval and decline rates by segment, measured against the champion rather than against a target.
• Score drift and feature drift on the challenger’s inputs, since a new model often leans on features the champion ignored and therefore never monitored.
• The false-positive budget: how many good applicants the challenger declines that the champion would have booked, priced against the losses it avoids.
• Adverse action reason codes, so a promoted model can still generate compliant, accurate declines under ECOA.

Lineage, audit trail, and the promotion gate

Every shadow score needs to be reproducible. Log the model version, the feature values as served, the feature store snapshot or timestamp, and the resulting decision, all keyed so you can join back to the eventual outcome. When a validator or an examiner asks why the challenger declined a specific applicant eight months ago, you reconstruct it from the log rather than a rerun against today’s data that would quietly give a different answer.

Promotion is a governance event, not a deploy. Treat the switch from champion to challenger the way you treat any other change to credit policy: versioned, reviewed, and reversible if the numbers turn against you. The gate should demand matured-outcome performance that clears a pre-registered bar, drift inside tolerance, adverse action codes that hold up, and sign-off from model risk. Under SR 11-7 the challenger enters scope the moment it influences a live decision, so the effective-challenge and documentation work belongs at the promotion gate rather than deferred until after go-live.

Keep the old champion warm after you promote. Demote it to shadow instead of deleting it, so if the new model drifts or a segment degrades you have an already-validated fallback and a live baseline to measure against. Champion/challenger is not a bake-off you win once and walk away from. It is how a lending model lives in production for good: one model deciding, another watching it, and a reproducible log tying the two together so you can always answer for a decision after the fact.