Building a RAG evaluation harness for finance

A RAG evaluation harness for finance is a repeatable test that scores two things separately: whether retrieval found the passage that answers the question, and whether the generated answer is grounded in what was retrieved. You build a labelled question set from real documents, run the pipeline against it on every change, and watch one headline number move before anything ships.

The reason to separate those two measurements is that they break for different reasons and you fix them in different places. A drop in retrieval quality is an indexing, chunking, or ranking problem. A drop in faithfulness with retrieval holding steady is a prompt or model problem. Collapse them into one “accuracy” score and you lose the ability to tell which half to go fix. Most teams find this out the slow way, after a confidently wrong answer at quarter-end that nobody can explain.

Measure retrieval on its own

Retrieval either put the supporting passage in the context window or it did not. Grade that first, before the model writes a word, because generation cannot recover from context that never contained the answer.

Build the eval set from real questions against real documents: 10-K sections, covenant schedules, transaction narratives, policy PDFs, whatever your system actually reads. For each question, a human marks which chunk or chunks contain the answer. That labelling is the expensive part and it is the part you cannot skip. Then track:

• Recall at k. Of the passages that truly support the answer, how many landed in the top k retrieved? This is the number that matters most, because a passage the model never sees cannot ground an answer.
• Precision at k, or context density. How much of what you retrieved is actually relevant? Padding the window with near-misses raises cost and gives the model more room to anchor on the wrong sentence.
• MRR or a rank-aware score, so you notice when the right chunk slips from position two to position eight even though recall looks flat.

Segment every one of these by document type and by question type. A harness that reports a single blended recall of 0.86 is hiding the fact that it retrieves filings well and covenant tables badly. The blended number moves a little; the covenant slice is the one that will hurt you.

Grade generation against the context, not the truth

Once the right passage is in the window, the second question is whether the answer stays inside it. This is faithfulness, and it is the expensive failure in finance. A clumsy sentence gets caught in review. A fluent sentence carrying a number the source never supported can run straight through to a client deck before anyone checks it against the filing.

Score generation on three axes, each against the retrieved context rather than against some external ideal:

• Faithfulness. Is every claim in the answer entailed by the retrieved passages? An answer that invents a figure, or restates a covenant threshold slightly off, fails here even if it reads well.
• Answer grounding. Do the citations point to passages that genuinely support the specific claim they are attached to? A citation that merely lands on the right page is not grounding; it has to support the sentence.
• Answer relevance. Did it address the question that was asked, or a nearby one it found easier to answer from context?

For faithfulness at scale we use an LLM judge, but only after calibrating it. Take a few hundred answers, have a human label each as grounded or not, then run the judge against that same slice and measure agreement. If the judge disagrees with humans a fifth of the time, its scores are noise and you tune the rubric until agreement holds. Report the judge’s agreement rate alongside its faithfulness scores. A faithfulness number from an uncalibrated judge is a vibe wearing a decimal point.

Wire it into the pipeline so the number moves

An eval set that lives in a notebook someone runs by hand is a document, not a harness. The harness is the automation around it.

• Run the full suite in CI on every change to prompts, chunking, the embedding model, the retriever, or the base model. A prompt tweak that lifts one question type and quietly regresses covenant retrieval should fail the build, not surface in production.
• Pin one headline metric to the decision the system drives, the way you would for any finance model, and set a threshold the build has to clear. For a research assistant that is usually faithfulness; for a policy-lookup tool it is retrieval recall.
• Version the eval set and keep its results in the audit trail. When validation or an external reviewer asks how the system performed before a release, the answer is a stored run against a known set, not a memory of a good demo.
• Feed production failures back in. Every wrong answer a reviewer catches becomes a labelled case in the next run, so the harness gets harder exactly where the system got caught.

Watch for drift once it is live. Documents change format, a new filing template appears, someone swaps the embedding model for a cheaper one. Retrieval recall sags first, faithfulness follows, and without a standing harness the first signal you get is a user disputing a number. With one, the covenant slice drops half a point in CI and you catch it before it ships.

The point of all this is a number that moves for a reason you can name. When retrieval recall falls, you know to look at chunking. When faithfulness falls while retrieval holds, you know to look at the prompt or the model. That separation is the whole value of the harness, and it is what turns “it looks good” into something you can put in front of an auditor.