Draw the auto-decision line where the score separates good from bad cleanly enough that a machine decision costs less than a human one, and refer everything else. That gives you three bands: auto-approve and auto-decline, each sized from the score distribution and your loss tolerance, plus a referral band in between where the model is uncertain or a policy rule fires.
Most teams set the threshold backwards. They decide they want to automate 80% of volume, find a cutoff that hits 80%, then discover in the vintage curves six months later that the marginal approvals in that band default at three times the rate of the core book. The direction has to run the other way. You start from what the score can actually distinguish, and the automation rate falls out of that. The 80% is a result, and it is often lower than the number the business asked for.
Size the bands from the score, not the target
An auto-approve band is defensible when the bad rate inside it is low, stable across slices, and low enough that the cost of the occasional bad decision is less than the cost of routing every one of those applications to a human. That is three separate tests, and a single approval rate hides all of them.
- Rank the population by score and look at the bad rate in each decile or finer bucket. The auto-approve band is the top of that ranking where the bad rate stays under your appetite, not a round percentile you picked in advance.
- Check the band holds across slices. A cutoff that looks clean on the whole population can hide a segment where the same score maps to a much worse outcome: thin-file applicants, a particular acquisition channel, a single product variant. If the bad rate at a given score differs by segment, the band has to differ by segment too, or the segment goes to referral.
- Price the errors. An auto-approve false positive is a booked loss. An auto-decline false positive is lost margin plus, sometimes, an ECOA-relevant decision you now have to justify. These are not symmetric, so the two cutoffs are not symmetric either. We set a false-positive budget on each side and let it drive the width of each band.
The band edges are estimates, and estimates have error bars. Near a cutoff the score is doing its weakest work, because that is by definition where good and bad applicants overlap most. We widen the referral band around each edge rather than pretending the boundary is a clean line. Applications that sit within the confidence interval of the cutoff go to a human, because that is exactly the population where a small model error flips the decision.
Referral rules are policy, not a dumping ground
The referral band catches two different things and it helps to keep them separate. One is model uncertainty, handled by the band widths above. The other is policy: conditions where you have decided a human must look regardless of what the score says. These are hard rules, and they should live in the decision engine as explicit, versioned logic, not baked into a model weight where nobody can see them.
Typical policy referrals: a score that disagrees sharply with a bureau flag, an application where affordability rests on income the cash-flow pipeline could not verify against a full cycle, a value above a limit, a document that failed a consistency check, an applicant who appears in a prior-fraud or entity-resolution match. Each of these is a stated reason to override the auto path, and each one should be logged with the rule that fired so you can audit why an application was pulled.
Two failure modes show up here. The first is a referral queue that grows faster than the analysts can clear it, which quietly turns straight-through processing into a backlog with extra steps. The second is the opposite: rules written so loosely that almost nothing refers, and the manual review that governance was counting on never actually happens. Both are measurable. We track the referral rate per rule and the queue age, and treat a rule that never fires or one that fires on a third of volume as a defect to investigate, not steady state.
Capture the override, then close the loop
The point of keeping a human in the loop is wasted if the human’s decision evaporates into an unstructured note. Every manual outcome on a referred application is a labelled data point about exactly the population the model finds hardest, and it is the most valuable label you have. Capture it as structured data at the moment of decision.
- Record the analyst decision, the score at decision time, the rule or uncertainty that triggered the referral, the reason the analyst gave, and the analyst identity. This is the raw material for both the audit trail and the next model.
- Keep overrides as a distinct label. An analyst approving below the auto-decline band, or declining inside the auto-approve band, is telling you something the model missed or got wrong. Feeding those straight back into training without review lets one analyst’s judgement, or bias, propagate silently. We hold overrides out of the primary training set until they clear a review, and we watch override rates by analyst and by protected-class proxy so a skewed pattern surfaces before it reaches a model.
- Tie the referral outcome to adverse-action logic. When a referral ends in a decline, the reason codes have to reflect what actually drove it, including the manual factors. A human in the loop does not relax the notice obligation; it adds a source of reasons that has to be captured accurately.
Then close the loop with vintage data. The only honest test of where you drew the line is how the auto-approved band performs once the loans season. Watch the bad rate inside the auto-approve band against the referral band the model wanted to send to a human. If the referred applications you ended up approving perform as well as the auto-approved ones, the band is too narrow and you are paying analysts to review good business. If the bottom of the auto-approve band is drifting toward the referral band’s loss rate, the line has moved and the model is stale. That makes the threshold a live control. You monitor it with the same discipline you apply to any model in production, and you move it when the vintage data says the population underneath it has changed.
FAQ
How high can the straight-through approval rate go before it stops being safe?
There is no universal ceiling. The safe rate is whatever the score can separate cleanly at your loss tolerance, and it moves as the population and the model drift. We size the auto-approve band from the score distribution and the observed bad rate in each slice, then recheck it every time the model is retrained or the mix of applicants shifts.
Do referred applications need the same adverse-action treatment as declines?
A referral that ends in a decline does. The reason codes must reflect the factors that actually drove the outcome, including anything the analyst weighed during the manual review, so a referral is not a way to bypass ECOA notice requirements. We attach reason-code capture to the referral queue itself, not only to the automated decline path.
How do you keep analyst overrides from quietly retraining the model on their own bias?
Overrides are logged as a separate label with the analyst identity, the reason, and the score at the time, and they are held out of the primary training set unless they clear a review. We monitor override rates by analyst and by protected-class proxy so a pattern of overrides that lifts one group and not another is visible before it reaches a model.