Model selection usually starts with the wrong question. A team asks which model is best, looks at a public leaderboard, and picks whatever sits at the top this month. The trouble is that the leaderboard was not measured on your task. It does not know your filings, your reconciliation rules, or what a wrong answer costs you at quarter-end.
So we start somewhere else. Write down the task in plain terms, then build a small eval set from real examples of it. Fifty or a hundred cases is often enough to be useful: the input, the answer you would accept, and the answers you would not. That eval set is the thing you actually optimise against. Once it exists, model selection turns into an experiment you can run, rather than an opinion you have to defend.
What the eval set tells you that a leaderboard cannot
A leaderboard reports an average across tasks that are not yours. Your eval set reports how a model does on filings extraction with your messy PDFs, or on entity resolution across the vendors you actually use. Those are different questions, and they often have different winners.
The eval also catches the failures that matter in finance and nowhere else. A model that scores well on general benchmarks may still hallucinate a reason code, or miss a restatement, or quietly read a value that did not exist on the date in question. You only see that if your eval contains cases designed to provoke it. A generic benchmark never will.
Run the comparison once and you usually find the ranking is not what the leaderboard suggested. Sometimes a cheaper model is within a point of the frontier one on your task. Sometimes it is not close, and now you know that too.
The tradeoffs you are actually buying
The choice between open-weight and frontier models is rarely about raw quality alone. Once a few models clear your eval bar, the decision moves to four things:
- Cost. A frontier API per-token price that looks trivial in a demo becomes real when you are processing every filing in a coverage universe. LLM cost and latency tend to move together, and both scale with volume in ways a prototype hides.
- Latency. An interactive analyst tool and an overnight batch job have very different tolerances. A model that is too slow for a chat interface may be perfectly fine for a job that runs while everyone is asleep.
- Context length. Some workloads need a whole 10-K in the window at once; others are better served by retrieval over a long document than by a model that claims to swallow the entire thing.
- Data residency. If the data cannot leave a jurisdiction or a private network, a hosted frontier API may be off the table regardless of how it scores. An open-weight model you run on-prem keeps sensitive records inside your own boundary, which is sometimes the deciding constraint before quality enters the conversation at all.
This is where a small fine-tuned or open model can beat a frontier API outright. On a narrow, repetitive task, a smaller model tuned on your examples often matches the big one on your eval, runs faster, costs less per call, and can sit on hardware you control. Generality stops being worth paying for when the task is not general.
Lock-in and the silent-update problem
A frontier API is a moving target. The provider can change the model behind the same endpoint, and your eval scores can shift without a single line of your code changing. For a system that produces an audit trail, that is a real exposure: the model that generated an adverse action notice last quarter may not be the one answering today, and you may have no record of the swap. Pinning versions helps where the vendor offers it. Keeping your eval set in continuous use helps more, because it turns a silent update into a visible regression you can catch and act on.
Open weights remove that particular risk. The model does not change unless you change it, and you can reproduce a past result exactly. That reproducibility has its own cost in hosting and maintenance, so it is a tradeoff rather than a free win. We weigh it case by case.
Route the work, do not pick one winner
There is rarely a single best model for a whole pipeline, and you do not have to choose one. Different sub-tasks have different needs. A cheap, fast model can handle high-volume classification or extraction; a stronger model can take the harder reasoning steps; a model you host yourself can process the records that are not allowed to leave the building. Each leg gets the model that wins its own slice of the eval.
Routing like this is more moving parts to maintain, and we only do it where the eval shows it earns its keep. But it is usually how a real system ends up looking, because the alternative is overpaying a frontier model to do work a smaller one does just as well.
The method is the point. Start from the task, measure on data that looks like production, and let the eval decide. The leaderboard is somebody else’s benchmark. Yours is the only one that pays your bills.