Model routing and fallback for cost and reliability

Routing sends each request to the cheapest model that can answer it correctly, and falls back to another model when the first one fails or degrades. A cascade escalates on quality: a small model answers, and only low-confidence cases move up to a larger one. Fallback handles operational failure: a timeout or rate limit reroutes to a healthy provider. Most finance workloads need both.

The instinct when a workload goes live is to pick the strongest model and point everything at it. It works in the demo and it is simple to reason about. It is also the most expensive and least reliable design you can ship. The strong model is slow, it rate-limits under a quarter-end spike, and you are paying frontier prices to classify whether a document is an invoice or a remittance advice. That last task a much smaller model does correctly, and faster.

The point of routing is to stop treating a heterogeneous stream of requests as if every one deserved the same compute. In a finance operation the stream really is heterogeneous. Extracting fields from a clean structured statement is not the same problem as reconciling three sources that disagree, and neither is the same as drafting a narrative an approver signs off on.

Classify the request before you pick the model

Routing starts with a cheap decision about the request, made before any expensive model runs. You are answering two questions: how hard is this call, and how much does a wrong answer cost here.

Difficulty you can often read off the input without a model at all. Length, structure, the presence of tables, the number of source systems the answer has to touch. A single well-formed PDF is not the same job as a bundle of scanned annexes. Where a rule is not enough, a small classifier or a short local model gives you a difficulty score for a fraction of a cent.

Cost of error is where finance work differs from general chat routing. Some outputs are read once and discarded. Others enter a payment file, a general ledger, or a suspicious-activity report, and a wrong number there has consequences well past the API bill. You route those to a stronger model and to a heavier verification path regardless of how easy the input looks. Route on both axes:

• Low difficulty, low stakes: smallest model, no verification. Field extraction from clean documents, obvious classifications.
• High difficulty, low stakes: mid or large model. Summaries and drafts a human reviews before anything happens.
• Any difficulty, high stakes: strongest model plus an independent check on the numbers. Anything that writes to a ledger, moves money, or feeds a regulatory filing.

Keep the router itself boring. Rules where you can state them, a small model where you cannot, and every decision logged with a hash of the input so you can replay why a request went where it did. The router is on the critical path for every call, so it cannot be the slow or flaky part.

Cascade for quality, and know when it stops paying

A cascade runs the cheap model first and escalates only when the answer is not good enough. The engineering is entirely in the word “enough.” You need a signal that a small model got it wrong, computed without the large model you are trying to avoid.

Useful signals in a finance context are concrete rather than vibes. Does an extracted total match the sum of the line items. Does a stated balance tie to the source document. Did the model return valid JSON against the schema. Is the reported confidence, or the token-level probability on the fields that matter, below a threshold you set from an eval set rather than guessed. When the cheap answer passes these, you keep it. When it does not, you escalate.

The number that tells you whether the cascade earns its place is the escalation rate on real traffic. If nine in ten calls settle at the cheap tier, the economics are excellent. If half of them climb to the expensive model, you are paying for two inferences plus the check to get one answer, and you would be better off routing that class of input straight to the strong model. Measure this per input type, not in aggregate, because a healthy overall rate can hide one document type that escalates almost every time. That document type belongs on a direct route.

Tune escalation thresholds against a labelled eval set with a false-positive budget you have agreed with the business, the same way you would tune a fraud score. Escalating too eagerly burns the savings. Escalating too rarely lets wrong answers through. Neither is a setting you find by intuition.

Fallback keeps the workload running when a provider fails

Fallback covers the model that would have answered fine but never got the chance. The provider is rate-limiting you, a region is down, latency has blown past your budget, or a deploy on their side changed behaviour overnight. None of this is about which answer is better. It is about staying up.

Design it as plumbing:

• Set a hard timeout and a retry budget per call. When they are exhausted, move on rather than hanging the queue.
• Keep a second model, ideally from a different provider, that can serve the same request with the same prompt contract and schema. If a swap needs prompt surgery, it is not a fallback you can trust at three in the morning.
• Use a circuit breaker. After a run of failures, stop hammering the primary, send traffic to the secondary, and probe the primary quietly before returning.
• Degrade deliberately for the high-stakes path. If both models are unreachable, a payment-affecting call should queue for human handling, not guess. Silent best-effort is the wrong default when money moves.

The two designs compose. The router decides where a request should go, the cascade decides whether the cheap answer holds, and fallback decides what to do when the intended model will not respond. Keep them as separate layers so you can reason about each. And log the whole path for every request, because when a finance number is questioned weeks later the answer to “which model produced this and why” needs to come from a record, not a reconstruction. In a workload that touches ledgers and filings, that audit trail is part of the product rather than overhead you trim later.