AI vendor due diligence: the questions that matter

When you buy an AI vendor, you buy their model risk. Their training data, their labelling choices, their drift, and their failure modes become yours the moment their output touches a customer decision or a general ledger. Due diligence is the work of finding those inherited risks before contract, not after an examiner asks who validated the model.

Most AI procurement in finance is run like software procurement. Security questionnaire, SOC 2 report, pen test summary, DPA, sign. That covers whether the vendor can keep your data safe. It says nothing about whether the model is correct, whether its accuracy claim survives contact with your data, or what happens to your straight-through processing rate the week their model drifts. Those are the questions that decide whether the thing works, and they are the ones most checklists skip.

Treat the accuracy number as an unverified claim

Every AI vendor leads with a metric. Ninety-four percent accuracy on invoice extraction, a fraud model that catches most of what rules miss, a reconciliation tool that auto-matches the long tail. Take none of it at face value. Ask what the number was measured against.

• What was the evaluation set? A vendor’s demo accuracy is measured on their data, which was cleaned and labelled by the people who built the product. Your invoices, your counterparties, and your entity names look nothing like it.
• How were the labels created, and by whom? If a domain expert labelled the ground truth, the number means something. If it was labelled by the same model or by cheap annotation, the number is measuring agreement, not correctness.
• Was the split time-based? This is where finance ML quietly breaks. If the vendor trained and tested on a random split of the same period, they have almost certainly leaked future information into the past. Ask directly whether their eval respects point-in-time correctness and holds out a forward window. Lookahead in a backtest is the most common way a vendor’s headline number is a fiction.
• What is the false-positive budget? A fraud or AML model that flags twice what your investigators can clear does not catch more fraud in practice. It burns review hours and pushes real alerts to the back of the queue. The metric that matters is precision at the recall you actually operate at, on an alert volume your team can work through in a day.

The only accuracy number worth trusting is the one you generate yourself. Before signing, run the vendor on a held-out slice of your own historical data with known outcomes. A vendor who refuses a paid proof-of-concept on your data, or who wants to tune on your test set first, is telling you their general number does not transfer.

Trace how the output was produced

A finance decision has to be explainable to an approver, an auditor, and eventually a regulator. That means the vendor’s output cannot be a bare answer. It has to carry its provenance.

• Lineage. For any figure the model produces, can the vendor show which source document, field, or transaction it came from? A reconciliation match without a traceable pair of records is unauditable. Extraction without a pointer to the character span on the page cannot be checked at quarter-end.
• Entity resolution. If the product joins counterparties, accounts, or securities across systems, ask how it decides two records are the same entity. This is where silent errors live. A wrong merge moves an exposure onto the wrong obligor and nobody sees it until it is large.
• The audit trail. Does the system record the model version, the input, the output, and the human who accepted or overrode it, for every decision? If a decision from eight months ago cannot be reconstructed, you cannot defend it to an examiner. Model risk expectations assume you can rebuild any decision on demand, so write the requirement into the contract rather than hoping the vendor logs enough.
• Human override. When a reviewer corrects the model, where does that correction go? If it silently retrains the model, you have a governance problem. If it goes nowhere, you have a learning problem. You want it captured, versioned, and fed back on a controlled cadence.

Ask to see one real decision end to end during evaluation. Pick a hard case, follow it from raw input to final output, and check that every number in between has a source you can point to.

Model risk and lifecycle: what happens after go-live

The failure that costs you is rarely at launch. It is six months in, when the world the model learned has shifted and nobody was watching.

• Drift monitoring. What does the vendor monitor in production, and what do they show you? Input distribution shift, output distribution shift, and match or acceptance rates over time are the minimum. If their monitoring is a green status page, it is theatre.
• Retraining and change control. When does the model change? A vendor who silently ships a new model version can break your straight-through processing overnight and invalidate every validation you did. You want release notes, version pinning, and the right to test a new version before it reaches your production traffic.
• Feature and data dependencies. If the product runs on a feature store or third-party data feeds, those are dependencies you are inheriting. Ask what happens to model output when an upstream feed is late or missing at quarter-end. A model that degrades quietly on stale features is worse than one that errors loudly.
• Concentration and exit. If this vendor sits inside a critical process, DORA treats them as an ICT third-party provider. You need audit rights, incident notification timelines, and a genuine exit plan: can you get your data, your labels, and your decision history out, and can the process run without them while you replace it?

What sign-off should actually depend on

Reduce the whole exercise to a few pass conditions. You ran the vendor on your own held-out data and the accuracy held up at your operating point. Every output is traceable to a source. The audit trail reconstructs any past decision. You control when the model version changes. You have monitoring you can read and an exit you could actually execute. If any of those is missing, the gap does not disappear because you bought instead of built. It moves onto your model risk inventory with your name against it.