On-prem, VPC or API: hosting AI in regulated finance

Where the model runs decides who can see the data, which regulator has a claim on it, and what you prove in an audit. For most workloads the answer is a public API with zero-retention terms and a pinned region. Use a VPC deployment when data cannot leave your tenancy, and self-hosted open weights only when residency or latency forces it. The technical differences are smaller than the contractual ones.

That last point trips up a lot of teams. Engineers compare tokens per second and context windows. The compliance function compares data flows, subprocessors and the country the request lands in. Both are looking at the same three options, and they rank them differently. The job of an AI architecture is to make one choice defensible on both axes at once.

The three shapes, and what each one actually costs you

Strip away the marketing and there are three hosting shapes for a language model in finance.

• Public API. You send a prompt to a provider’s shared endpoint and get a completion back. It is the cheapest to start with and the fastest path to the best models, and it is also the one your risk team will question hardest. The prompt leaves your environment. Whether that is acceptable depends on the terms attached to the endpoint, not the endpoint itself.
• VPC or dedicated deployment. The provider runs the model inside your cloud tenancy, or gives you a single-tenant instance in a region you name. The data path stays inside a boundary you control and can point to on a diagram. You pay for that with higher cost, fewer model options, and a slower route to whatever the frontier lab shipped last week.
• Self-hosted open weights. You download the weights and serve them on your own GPUs, on-prem or in your own cloud account. Nothing about the request touches a third party. In exchange you own the whole lifecycle: patching, capacity, quantization choices, the eval harness, and every regression when you upgrade a checkpoint.

The instinct in a regulated shop is to reach straight for the third option because it feels safest. It removes one problem, the third-party data transfer, and hands you several others. A self-hosted model that nobody re-evaluates after an upgrade will drift on your workload silently. On-prem GPUs that sit idle outside quarter-end are expensive insurance. The safe-by-default framing hides the operational bill.

Start from the data, not the model

Before comparing hosting options, classify what actually goes into the prompt. The hosting decision follows from the data class, and it is usually finer-grained than “we work with financial data.”

• Is there personal data in the prompt, and under which jurisdiction? A GDPR data subject and a US consumer under GLBA pull in different residency and disclosure rules.
• Is the payload material non-public information, positions, or anything that would move a market if it leaked? MNPI often carries stricter internal handling rules than the law strictly requires.
• Does the data carry a residency obligation, whether from a regulator, a client contract, or your own policy, that names a region or forbids a transfer?
• What is the lineage story? If a completion ends up in a filed document or a customer-facing decision, you need to show where the input came from and reconstruct the exact request months later.

Once the data is classified, a lot of the debate collapses. Public MNPI-free reference data with no personal fields can go to a zero-retention API without much argument. A prompt containing customer PII under a data-residency clause that names Frankfurt cannot leave an EU boundary, which rules the shared US endpoint out on contract terms before you ever benchmark it. Model quality only matters among the options the data class already permits.

This is also where the audit trail gets designed, not bolted on. Every inference call in a regulated flow should log the prompt, the model and version, the region it executed in, and a hash that ties the completion back to the record it fed. When a supervisor asks how a straight-through decision was made in March, the answer has to be reconstructable without paging the on-call engineer.

DORA changed the calculus, and not in the direction people expect

Under the EU’s Digital Operational Resilience Act, an external model provider is an ICT third-party service provider. If the workload it supports is a critical or important function, the provider falls inside your ICT third-party risk framework: contractual clauses on subprocessors, audit and access rights, an exit strategy, and inclusion in your register of information. None of this makes the API a shortcut around governance. You manage the provider like any other critical dependency.

Two consequences follow that teams tend to miss.

• Self-hosting keeps you inside the framework. Bringing the model in-house removes the third-party transfer, but the operational resilience obligations now sit on you directly: capacity planning, incident response, resilience testing. You swap a supplier you must oversee for a system you must run. Neither is free.
• Concentration risk is real. If your fraud scoring, credit memos and customer service all route to one provider’s endpoints, that is a single dependency the regulator will notice. Part of the hosting decision is deciding whether a fallback path, a second provider or a self-hosted model behind a feature flag, is worth building before you need it.

None of this argues for one shape over another. It argues for making the choice per workload, from the data class up, with the contract and the audit trail treated as first-class parts of the design rather than paperwork you attach afterward. A fraud model that scores every transaction and a quarterly narrative-generation job pull in different residency exposure and very different latency budgets. One needs a sub-100ms answer on the payment path; the other runs overnight at quarter-end. They rarely deserve the same hosting answer, and forcing them onto one platform to keep the architecture tidy is how a defensible system turns into a fragile one.