An LLM gateway for access, cost and governance

An LLM gateway is a proxy every application calls instead of the provider directly. It holds the credentials, attaches an identity to each request, enforces quotas and policy, and writes an audit trail. You get one place to see and control what leaves the building. Most model spend and compliance exposure starts with the opposite: one team’s API key, shared across services and pasted into a notebook.

The reason this matters more in finance than in a general software shop is that the thing leaving the building is often customer data going to a third party, and the thing coming back can end up in a document an approver signs. Both directions need a record. Without a gateway you have neither the record nor the control, and you find out the shape of your exposure when finance asks why the model bill tripled at quarter-end, or when a reviewer asks which requests contained account numbers.

What sits between your apps and the providers

The gateway is a reverse proxy that speaks the provider APIs on the outbound side and a normalized API on the inbound side. Applications authenticate to it with their own short-lived credentials. They never see the provider keys. Those live in a secrets manager and rotate on a schedule the applications do not have to know about.

Concretely, a request arrives, and before it goes anywhere the gateway does a fixed sequence:

• Resolve the caller to an application identity and a team, so cost and access decisions have a subject.
• Check the request against policy: is this app allowed to call this model, with this data class, for this declared purpose.
• Apply quota and rate limits at the identity level, and reject or queue when a budget is exhausted rather than letting the call through.
• Rewrite the request onto whichever provider and model the routing rules select, and inject the real credential.
• Stream the response back while recording the full exchange and the token counts the provider returns.

Provider abstraction falls out of this design almost for free, and it is the least interesting benefit. Nobody switches vendors on a whim. What the normalized API buys you is a single place to react when a provider deprecates a model, reprices it overnight, or goes down in the middle of your busiest reconciliation window. You change a routing rule instead of editing and redeploying every service. Failover from a primary model to a fallback becomes a config change with a clear audit record of when it happened and why.

Cost controls that bite before the invoice does

Model spend is unusual because a single misconfigured loop can generate a five-figure bill overnight, and nothing stops it until the credit card does. The gateway is where you put the brakes, because it is the only component that sees every call and can attribute each one.

Total spend is the wrong number to watch. What you meter is spend per identity per unit time. The gateway counts tokens as they are billed, tags every call with the application, team and purpose, and enforces budgets at that granularity. A batch entity-resolution job that normally processes ten thousand records gets a ceiling; when a bad query fans it out to ten million, the job hits its budget and stops instead of running until someone notices.

• Hard quotas per identity, so one team cannot spend another team’s headroom.
• Soft alerts before the hard limit, routed to the owning team, not to a generic ops channel nobody reads.
• Per-purpose attribution, so finance can see that document drafting cost one thing and transaction monitoring cost another, and price the internal chargeback accordingly.
• Model-tier policy, so a summarization task cannot silently be pointed at the most expensive model because a developer copied a snippet.

The attribution is what turns cost from a monthly surprise into a line item somebody owns. It also feeds capacity planning, because you now have real per-workload demand instead of a single opaque provider bill.

The audit trail and the policy layer

For a regulated user the logging is the reason the gateway exists at all. Every request and response is recorded with the identity, the declared purpose, the model, the token counts, and a classification of the data that went out. Retention is applied at write time according to the data class, so a request that carried personal data is governed by the rule for that class, not by whatever default the log store happens to have.

This is what makes the third-party question answerable. Sending customer data to an external model is a processor relationship, and under DORA the provider is an ICT third party you are expected to monitor and be able to exit. The gateway gives you the monitoring surface: what went to whom, when, and whether it should have. When you need to cut a provider off, you do it at the gateway and the calls stop everywhere at once.

Policy sits at the same chokepoint. Because every call passes through, you can enforce rules that no individual application can be trusted to enforce for itself:

• Block or redact requests whose data classification exceeds what a given provider is contractually allowed to receive.
• Deny models or regions that fail a residency requirement, before the request leaves your network.
• Require a purpose tag on every call, and reject untagged traffic, so nobody can send data to a model without saying why.
• Keep a policy version on each logged call, so a reviewer can see which rules were in force when a decision was made.

None of this removes the need for careful application design. A gateway does not know whether a model’s output has a wrong number in it. What it gives you is the one place where access, spend and evidence are controlled together. When someone asks what your systems sent to which model and what it cost, you answer from your own records instead of opening a ticket with a vendor’s support desk.