Most private LLM programmes stall at the same point. The pilot works, a single GPU box serves one team well enough, and then someone asks how a second business unit gets access, how prompts are logged for audit, and what happens when the node reboots. At that point the question stops being about models and becomes an infrastructure question that the AI team is not usually staffed to answer.
The useful framing in 2026 is that there is no single private deployment. There are three topologies, and they differ in who holds the hardware, not in what the software does. On-premises means the model, the data, and the serving run on machines you own inside your own network boundary. Private cloud, usually an isolated VPC, means inference runs in cloud infrastructure under your IAM, network controls, and logging, with no shared tenancy on the inference path. Hybrid means both, with a routing layer deciding which requests go where.
All three sit under the same control plane: an API gateway that handles authentication and routing, observability that captures prompts, responses, and GPU metrics, and a retrieval tier if your use case involves internal documents. Getting that control plane right matters more than the topology choice, because it is what lets you change the topology later without rewriting every application that consumes the models.
The three topologies and what each one costs you
On-premises gives you physical custody. Nothing leaves the building, which is the only acceptable posture for air-gapped, export-controlled, and defence-adjacent work, and increasingly for organisations whose legal position on data residency will not tolerate a cloud provider in the chain. The cost is that you now own hardware procurement, capacity planning, patching, high availability, and GPU operations. Those are real disciplines with real headcount attached, and they are the reason on-prem pilots often outperform on-prem production.
Private VPC is the enterprise default in practice, and for good reason. You get isolation, your own logging and identity boundary, and elastic capacity that does not depend on a procurement cycle measured in months. You give up physical control, and you take on the work of proving tenant isolation and egress control to your own auditors. For most regulated enterprises this trade is correct.
Hybrid is where mixed-risk organisations end up. Sensitive workloads stay private, burst and low-risk traffic can go elsewhere, and the routing policy becomes a governed artefact. The cost is complexity in exactly the place where mistakes are expensive: data classification and policy enforcement at the gateway.
A fourth pattern is worth naming separately because it saves real money. Split-tier inference keeps small, high-volume models on private infrastructure for first-pass work such as classification, extraction, and routing, and reserves large models for the small fraction of requests that need heavy reasoning. Most enterprise token volume is not hard. Sizing your private tier for the hard 5% is how private AI ends up looking more expensive than it is.
The reference stack, layer by layer
A production private stack has five layers, and each has a small number of choices that actually get made.
- Infrastructure. GPU servers or a GPU cluster, with networking and storage paths sized for GPU traffic rather than general workloads. This is where most first-time deployments underspecify, because attention goes to the cards and not to what feeds them.
- Serving runtime. vLLM is the common choice for high-concurrency multi-user serving, largely because of how it handles KV cache under load. TensorRT-LLM is the option when you want maximum efficiency from NVIDIA hardware and are willing to accept a heavier build and optimisation step. SGLang fits when execution orchestration, structured output, or agentic control flow is the harder part of the problem. Ollama and llama.cpp remain reasonable for single-user and edge cases, but they are not the shape of a shared production tier.
- Gateway and security. A reverse proxy or API gateway (NGINX, Envoy, Traefik, Caddy) terminates TLS and handles auth, routing, and rate limiting. Authentication integrates with what you already run: Active Directory, Okta, or Entra ID. Network segmentation and dedicated VLANs are standard in hardened deployments and cheap to do at build time, expensive to retrofit.
- Observability. Prompt and response logs, GPU utilisation, latency percentiles, queue depth, and error rates, exported to whatever your enterprise already uses. Prometheus and Grafana are the common pairing. Treat this as a production requirement rather than a phase-two item, because without it you cannot answer capacity questions or incident questions.
- Retrieval. For knowledge use cases, a vector store such as Milvus, Qdrant, Weaviate, pgvector, or Chroma sits beside the inference tier. Keep it architecturally separate from serving. Retrieval is where document-level access control gets enforced, and it scales on a different curve than inference.
Multi-tenancy: pick your blast radius
Once a second team wants access, you make an isolation decision whether or not you notice.
- Shared cluster with isolated namespaces is the most efficient use of GPUs and the hardest to secure correctly. Isolation comes from Kubernetes namespaces, network policies, per-tenant authentication, and quota enforcement. It works when you have a platform team that already operates multi-tenant infrastructure.
- Shared control plane with dedicated data plane gives tenants a common gateway, observability, and orchestration while each gets its own inference nodes or node pool. This is the usual compromise, and it is what we recommend when business units have different data classifications but the same operational owner.
- Dedicated per-tenant stack means separate clusters, storage, and logs. It costs the most and is the right answer for separate legal entities, for a regulated division inside a wider group, or where a single tenant’s compliance scope would otherwise contaminate everything else.
Whichever you choose, the minimum controls are the same: authentication, authorisation, per-tenant rate limits, logging, and network segmentation. Quota enforcement deserves specific attention, because on shared GPUs one team’s batch job is another team’s latency incident.
What the gateway actually decides
The LLM gateway is the piece most teams build last and should build first. It routes by model, by tenant, by prompt sensitivity, by latency tier, and by fallback policy. Around that, production deployments accumulate prompt templates, retrieval pipelines, semantic caches, guardrail and DLP checks, and per-application budget enforcement.
The reason to put it in early is that it is the only place where policy is enforceable rather than advisory. Sensitivity classification, model allowlists, and egress rules live there. If applications call inference endpoints directly, every new application becomes a separate compliance review, and changing model or topology means changing every caller.
How to decide, and where to start
Work through the criteria in order. Data sensitivity and regulatory scope come first: if the environment must be disconnected or export-controlled, you are on-prem and the rest of the decision follows from that. If compliance is required but cloud controls satisfy your regulator and your legal team, private VPC will get you to production faster. Then check operational maturity honestly. If you do not have GPU operations, observability, and platform engineering capacity today, on-premises will consume the programme before it delivers anything.
Cost and utilisation come next. On-prem is cheaper at sustained high utilisation, but only after you absorb procurement lead times and operations overhead, which means it rewards steady load and punishes uncertain demand. Availability is the last gate: two inference servers behind a load balancer is the baseline, and anything less is a pilot regardless of what you call it.
In our experience the fastest credible start is a private VPC deployment with vLLM behind a gateway, one vector store, and full telemetry from day one, sized for your actual first-pass workload rather than your most demanding one. Get that into production with real users, measure token volume and latency by application, and let those numbers drive the on-prem decision. Buying hardware before you have production traffic data is how organisations end up with idle GPUs and a model tier that does not match what people ask of it.
FAQ
Is on-premises always the right answer for regulated data?
No. On-premises is the right answer when the data cannot leave your physical boundary, when you are export-controlled, or when the environment is genuinely disconnected. For most regulated workloads, an isolated cloud VPC under your own IAM, network policy, and logging gives you the compliance posture you need with far less operational load. The deciding question is not sensitivity in the abstract but whether your legal or security position requires physical custody of the hardware. If it does not, private VPC is usually the faster route to production.
Do we need Kubernetes to run a private LLM stack?
Not on day one. A single GPU node running vLLM behind a reverse proxy is a legitimate production deployment for a single team, and we have put that shape into service more than once. Kubernetes earns its keep when you have several models, several tenants, rolling upgrades, and node-level failure to plan for. If your platform team already runs Kubernetes, use it. If not, adopting an orchestrator and an inference stack at the same time doubles the number of things that can fail during your first rollout.
How do we handle requests that genuinely need a frontier model?
Route them, explicitly. The pattern is a gateway that classifies every request by data sensitivity and task type, keeps regulated and high-volume traffic on private infrastructure, and forwards a defined, allow-listed subset to a managed API. The control plane needs data-classification rules, model allowlists, and egress controls so the routing decision is enforced rather than documented. What breaks this pattern is an implicit fallback: if overflow traffic silently reaches a public endpoint during a capacity incident, your data boundary is decorative.