Eval-driven development means you write the scored test before you build the feature. You define a set of graded cases that encode what a correct answer looks like, wire them into CI, and let them gate every change to the AI path. A pull request that lowers the score does not merge. The eval set is the contract; the code has to satisfy it.
The reason to work this way in finance is that the usual signals of quality lie to you. A model that reads a bank statement and returns a confident, well-formatted, wrong income figure looks fine in a demo and fine in a manual spot-check. It fails on the twelfth statement type nobody thought to try. Without a standing eval set, you find that failure in production, inside a straight-through decision, after an auditor asks how the number was derived. The eval moves that discovery to the left of the ship line.
Build the eval set before the feature
The eval set is a versioned artifact that lives in the repository next to the code it grades. Each case is an input, an expected outcome, and a grader that decides pass or fail. You write it first because the act of writing it forces you to state what the feature is supposed to do, in terms specific enough to score.
Sourcing the cases matters more than the count. Pull from three places:
- Real production traces, sampled across document types, obligors, and reporting periods, so the distribution matches what the system actually sees rather than what is convenient to test.
- Past incidents. Every wrong number that reached a human becomes a permanent case. This is the regression eval; once a bug is in the set, it cannot come back silently.
- Adversarial cases you construct on purpose: the amended filing, the statement with a lookahead trap where a later page contradicts an earlier one, the name that resolves to two different counterparties.
Point-in-time correctness deserves its own attention. If your eval feeds the model data that would not have existed at the decision date, you are testing against leakage and your scores are inflated. Freeze each case to the information available at its as-of date, and grade the answer against what was knowable then.
Grade with the strictest method the task allows. A computed figure gets an exact or tolerance-based numeric check. A citation gets a resolver that confirms the referenced document and line actually exist and say what the model claims. Reserve LLM-judged rubrics for genuinely graded qualities like completeness or tone, and only after you have checked the judge’s agreement with human labels on a holdout. An unvalidated judge is a random number generator with good grammar.
Gate every change in CI
Once the set exists, the eval runs in continuous integration on every pull request that touches the AI path: prompts, retrieval, chunking, model version, tool definitions, post-processing. The job scores the current build and compares it against frozen thresholds per metric. Fall below threshold on any protected metric and the check fails, blocking the pull request exactly as a failing unit test would.
A few practices keep this honest.
- Separate the metrics rather than blend them into one number. Track exact-match accuracy, citation validity, refusal rate on out-of-scope inputs, and your false-positive budget independently. A blended score lets a gain in one place hide a regression in another.
- Pin the model and decoding parameters, and record them in the run. When the score moves, you need to know whether the code changed or the vendor did. Bound non-determinism with fixed seeds where they exist, and by running to a stable aggregate where they do not.
- Keep a holdout slice the developers writing prompts never see. Tuning against every visible case eventually overfits the prompt to the eval, and the holdout is how you catch that.
Set thresholds conservatively. You do not pin them to the current score and forbid all movement; you set a floor that protects the behaviors you have promised and let ordinary noise pass. Raising a threshold is a decision made in review, with the trace lineage showing which cases moved and why.
Treat the eval as a governed artifact
An eval set that gates production decisions is itself part of the audit trail. Version it, review changes to it in pull requests, and require a reason when a case is edited or removed. Deleting an inconvenient case or quietly lowering a threshold is the AI equivalent of editing the test to make the build go green, and it should be as visible as any other control change. Under a model-risk framing like SR 11-7, the eval set is a large part of what “ongoing monitoring” and “effective challenge” concretely look like for a language model, and DORA’s resilience-testing expectations for critical systems point the same way.
The payoff compounds. Every incident that becomes a case makes the next regression cheaper to catch. Every new document type sampled into the set widens what the gate protects. Over a quarter, the eval set becomes the most accurate description you have of what the system does, drift and all, because it is the one description that gets executed on every change instead of written once and left to rot. When an auditor asks why you trust a model to touch a general ledger or an adverse-action decision, the eval set is the answer, and its version history and record of blocked merges are the evidence behind it.
FAQ
How many cases do we need before an eval set is useful?
Fewer than teams expect. Thirty to fifty cases per behavior, drawn from real production traces and past incidents, catches most regressions. Precision of the labels matters far more than volume; a hundred sloppily graded cases are worse than forty clean ones.
Can an LLM grade the eval, or do we need human labels?
Both. Use human-labeled expected answers for anything with a defensible ground truth, such as a computed figure or a citation that must resolve. Use an LLM judge only for graded rubrics like tone or completeness, and validate the judge against human labels before you trust its score.
What blocks a merge when the eval regresses?
A drop below the frozen threshold on any protected metric fails the CI job the same way a failing unit test does. The pull request cannot merge until the score recovers or the threshold is deliberately renegotiated in the review.