Model risk management has a settled vocabulary. SR 11-7, the 2011 supervisory guidance from the Federal Reserve and the OCC, built the discipline on four pillars: validation, documentation, governance, and ongoing monitoring. For a logistic regression scoring credit applications, those pillars are well understood. You can write down the inputs, re-derive the output by hand, and back out exactly why a decision came out the way it did.
An LLM extracting figures from filings does not behave like that. Examiners have noticed. The common finding now is a governance gap: institutions shipped ML faster than their MRM frameworks adapted, so the model inventory has entries that the validation process was never designed to test.
In April 2026 the Federal Reserve, OCC and FDIC jointly issued SR 26-2, which supersedes SR 11-7. It modernises model risk management around a materiality construct, scaling the depth of oversight to a model’s actual risk and retiring the blanket expectation of annual revalidation. What it pointedly does not do is tell you how to validate a generative model: SR 26-2 excludes generative and agentic AI from its scope, calling the technology too novel and fast-moving to pin down. So the four pillars still stand, the framework around them is newer, and the hardest case, the LLM itself, is left to your judgement. That is the gap this note is about.
This is a method note, not legal advice. Here is what actually changes in the work.
Validating a system that will not give the same answer twice
Classic model validation leans on reproducibility. Run the same input, get the same output, compare against an independent benchmark. A non-deterministic system breaks that assumption at the first step. Temperature, sampling, and a vendor’s silent model update all mean today’s output may not match yesterday’s.
So validation moves from checking a single answer to characterising a distribution of answers. We hold a fixed eval set, run it repeatedly, and measure how stable the outputs are, not just whether one run looked right. For an extraction task that means scoring against a hand-labelled ground truth at quarter-end conditions, with the failure modes named: hallucinated line items, transposed figures, the wrong fiscal period pulled from a document that contains several. A single passing run tells you almost nothing. The variance across runs is the thing being validated.
Documentation has to reach the data and the prompt
SR 11-7 documentation was about model logic and assumptions. With ML the sensitive surface moves outward, to the data and to the instructions.
Two artefacts that older MRM templates rarely asked for:
- Data lineage for the training and retrieval corpus: where each document came from, when it was current, and whether point-in-time correctness held so the model was never exposed to information from after the decision date
- Versioned prompts and retrieval configuration, treated as model inputs, because changing a prompt changes the model’s behaviour as surely as changing a coefficient, and an audit trail that omits it cannot reconstruct a past decision
If you cannot answer “what exactly produced this output last March,” the documentation pillar is not satisfied, whatever the model card says.
Monitoring drift, and the vendor you cannot see inside
Ongoing monitoring used to watch input distributions and performance against realised outcomes. That still applies. Drift in the underlying population shows up the same way it always did. What is new is that the model itself can drift underneath you when it is hosted by a vendor who reserves the right to update weights without notice.
We treat that as an operational reality to instrument, not argue with. A standing eval set runs on a schedule against the live endpoint, so a regression caused by a provider-side change surfaces as a number rather than a support ticket weeks later. Where a vendor model is genuinely opaque, the validation effort shifts to the boundary: stress the inputs, bound the outputs, and reconcile against an independent source the model never saw. You are validating the wrapper and the controls around the black box, since the box itself will not open.
None of this replaces the SR 11-7 pillars. It extends them to systems that are probabilistic, partly external, and changing under production load. The teams that struggle are the ones still validating ML as if it were a spreadsheet. Start by writing down what your current process assumes about determinism, then test each assumption against the model you are actually running.