Red-teaming a finance AI system means attacking it on purpose, in a controlled harness, to find inputs that make it leak data, bypass its guardrails, call a tool it should not, or produce a biased decision. You run the attacks an adversary and an examiner would run, log what breaks, and fix the path before either finds it. It is adversarial testing, held to the evidence standard of model validation.
Most teams test whether their model gives good answers to reasonable questions. That is the eval set, and it is necessary. It tells you nothing about what happens when someone feeds the system a customer email with an instruction buried in it, or asks the same denied question fourteen different ways until one phrasing slips through. The interesting failures live in inputs nobody on the build team would think to type, which is exactly why you have to manufacture them.
What you are actually attacking
The model is one component. The thing that can hurt you is the application around it: the retrieval corpus it reads, the tools it can call, the records it can reach, and the workflow that acts on its output. A jailbreak is only a security event when it lets the model do something downstream. So the harness targets the whole path.
Concretely, in a finance deployment, we build attack sets against each of these:
- Jailbreaks and instruction override. Prompts that try to dislodge the system prompt: role-play framings, fake system messages, hypotheticals, encoded or translated payloads, and the long tail of published jailbreak templates. The question is whether any of them get the model to answer something policy forbids, such as revealing another customer’s data or explaining how to structure transactions under a reporting threshold.
- Prompt injection through data. The dangerous case in a retrieval or agent system. An instruction planted in a document, an email, a KYC attachment or a memo field that the model later reads as if it were a command. If the model can call tools, injected text that says “export this counterparty’s exposure” is the attack that keeps you up at night.
- Data exfiltration. Attempts to pull training data, other tenants’ records, prompt contents, or internal identifiers back out through the output. In a multi-tenant finance product this is where entity resolution and access scoping get stress-tested: can a crafted query make the system return a record the requester has no right to see?
- Bias and disparate outcomes. For any model touching credit, pricing or account actions, adversarial inputs that probe whether protected-class proxies shift the decision. This is not a nice-to-have. Under ECOA a credit decision cannot rest on a prohibited basis, and a red-team suite that varies proxy features while holding the real signal constant is how you find a proxy leak before an examiner does.
Each category becomes a corpus of concrete inputs with an expected safe behaviour attached. That is what makes it a test and not a demo.
Building the attack set and the harness
Automated generation gets you volume. Human adversaries get you the attacks that matter. You want both, and you want them running against the deployed configuration, not a notebook.
We usually start by seeding from known material: the public jailbreak and injection collections, prior incidents, and anything the client’s own logs already show people trying. Then we mutate. Paraphrase, encode, translate, split an instruction across turns, wrap it in the formatting of a real document the system ingests. A single seed attack becomes hundreds of variants, and coverage of phrasings is the whole point because the model’s refusal is not stable across them.
The harness itself has a few non-negotiable properties:
- It hits the real application path, including retrieval and tool calls, so an injected instruction actually reaches whatever it would reach in production.
- Every attack carries a machine-checkable pass or fail condition. “Did the model refuse” is judged by a grader, often a second model plus deterministic checks on whether a forbidden tool fired or a scoped record appeared in the output. A human spot-checks the grader on a sample, because a grader that quietly mislabels is worse than no grader.
- Results are logged with the exact input, the model version, the retrieval snapshot and the tool trace. When something breaks you need to reproduce it, and quarter-end is a bad time to discover you cannot.
Attack success rate per category is the headline number, but the useful output is the set of inputs that got through. Those go straight into a regression suite so the same attack cannot pass twice.
Turning findings into controls and evidence
A red-team finding is a bug with a blast radius. Triage it the way you would a security defect: what did it let the model do, how far downstream could it propagate, and what stops it. The fixes are rarely a better system prompt alone, because prompt-level defences erode as attackers rephrase. Durable controls sit in code around the model:
- Scope every tool call and every retrieval to the requester’s entitlements, checked deterministically, so a jailbroken model still cannot reach a record the user is not entitled to.
- Treat retrieved and user-supplied text as data, never as instructions, and validate the model’s output against source records before any figure or action reaches straight-through processing.
- Put a false-positive budget on your input filters. Blocking is cheap until it blocks a legitimate analyst forty times a day, at which point people route around the control and you are worse off than before.
Then keep the whole thing running. The attack corpus is not a one-time exercise. It grows every time production surfaces a new phrasing and every time you wire in a new tool. Run it in CI on every change to the prompt, the model version, the retrieval corpus or the tool set, and on a cadence in between.
The audit trail matters as much as the defence. Under SR 11-7 a model in a decision path has to be monitored and its risks documented, and DORA pulls the whole ICT stack around it into resilience testing. A versioned adversarial suite, with attack success rates over time and a record of each finding and its fix, is a large part of how you show a regulator that the system was tested against abuse and not just against the happy path. It is also, on the day something does get through, the difference between a logged finding you already remediated and an incident you are explaining after the fact.
FAQ
How is red-teaming different from a normal eval set?
An eval set measures accuracy on inputs you expect. Red-teaming measures behaviour on inputs an adversary crafts to break the system, so the two run against different corpora and answer different questions. You need both, and the red-team corpus grows every time production surfaces a new attack.
Do we red-team the model or the whole application?
The application. A jailbreak only matters if it lets the model reach a tool, a record or a decision it should not. Attacks that route through retrieval, tool calls and downstream systems are where finance risk actually lives, so the harness has to exercise the full path, not just the prompt.
How often should a deployed finance AI system be re-tested?
On every change to the prompt, model version, retrieval corpus or tool set, and on a fixed cadence between changes. Under SR 11-7 a model in a decision path is subject to ongoing monitoring, and an adversarial suite that runs in CI is a large part of how you evidence it.