Reject inference done responsibly

Reject inference is the set of techniques you use to estimate how loans you declined would have performed, because you only observe repayment for the applications you approved. Done well, it is a way to reduce selection bias in a scorecard. Done carelessly, it manufactures agreeable outcomes for the exact borrowers your current policy already distrusts, and it flatters every metric you report afterward.

The problem is structural. Your training data samples the applicants who cleared a prior model, a policy layer, and often a human underwriter, never the full population that applied. The rejects are missing not at random. They are missing precisely because someone predicted they would default, which means the correlation between your features and repayment inside the approved set is not the correlation in the population you actually score at the top of the funnel.

What reject inference is really asking

When you fit a scorecard on approved-and-observed loans, you are estimating P(default | features, approved). What you deploy against is P(default | features) for everyone who applies. The gap between those two is the selection effect, and its size depends on how aggressive your prior cutoff was and how much unobserved information the prior decision used.

If the prior decision used only variables you still have in the feature store, the selection is on observables and a well-specified model handles it. The trouble is the residual. Underwriters see documents, call references, notice things that never land in a table. Any signal that drove the historical approval but is absent from your features shows up as selection on unobservables, and no reweighting of the approved data recovers it. This is the same reason a model can look calibrated in backtest and drift the moment it meets applicants the old policy would have turned away.

So reject inference is an assumption you are making about the unobserved, dressed up as a data step. That is fine, as long as you name the assumption and check it where you can.

The methods, and where each one lies to you

The common techniques are not equally honest. Here is how we rank them and what each one quietly assumes.

• Random-approval holdout. Approve a small, deliberately chosen band of applicants below your cutoff and book them at controlled exposure. You now have real outcomes for rejects. This is the only method that produces evidence rather than assumption. It costs money in expected losses, so you size the band against a loss budget and concentrate it near the cutoff where the model is most uncertain.
• Reject purchase. Buy bureau performance on declined applicants who borrowed elsewhere. You get outcomes, but on a self-selected subset who found credit somewhere, and their behavior on someone else’s product at someone else’s price is not your counterfactual. Useful as a sanity check, weak as ground truth.
• Heckman two-stage. Estimate an approval model and a repayment model jointly, correcting the second for selection into the first. Clean in theory. In practice it needs an exclusion restriction, a variable that moves approval but not repayment, and credit rarely offers one that survives scrutiny. Without it the correction is identified only off the functional form, which is a polite way of saying it is guessing.
• Parcelling and fuzzy augmentation. Score the rejects with the current model, assign inferred good/bad status by their predicted probability, and fold them back into training with weights. This is the most popular method and the most circular. You are teaching the new model to agree with the old model on exactly the cases where you have no way to check. It stabilizes coefficients and it can entrench a bias for years.

We use parcelling when nothing better is affordable, and we treat its output as a regularizer, not as truth. The default we push for is a random-approval holdout sized to a loss budget, because it is the only design where the model gets contradicted by reality instead of by its own predecessor.

Keeping yourself honest in validation

The failure mode is not the inference step. It is reporting validation metrics as if the inferred outcomes were observed. If your AUC, KS, or calibration is computed on a population padded with model-assigned labels, those numbers describe the model’s agreement with its own assumptions.

What we hold to:

• Compute all headline performance on the approved-and-observed population only. Report the approval rate alongside it, so a reader knows what share of the funnel the metric actually covers.
• Track outcomes from any random-approval band separately, and compare the model’s predicted default rate on that band to the realized rate. This is the check that catches selection on unobservables, because those applicants entered without the old policy’s veto.
• Keep point-in-time correctness across the whole exercise. Reject inference multiplies the surface for lookahead: it is easy to infer a reject’s status using bureau data timestamped after the original decision, which leaks future information into a historical label. Every inferred outcome needs the same as-of discipline as a real one, with lineage back to the snapshot it was built from.
• Version the inference assumptions in the model’s audit trail. When the scorecard goes to validation under an SR 11-7 style review, or the decision is contested, someone needs to reconstruct not just the features but which rejects were inferred, by what method, and under what weight.

Reject inference does not remove selection bias. It substitutes an explicit, checkable assumption for an implicit, invisible one, and that trade is only worth making if you keep the assumption visible. The moment the inferred labels stop being flagged as inferred, you have built a model that measures its own confidence and calls it accuracy.