Private AI vs API: the actual cost crossover

The cost argument for self-hosting language models is made confidently and usually wrongly. A team compares an hourly GPU rate against a per-million-token API price, finds the GPU cheaper at full utilisation, and concludes the case is settled. It is not settled, because full utilisation is not what production traffic looks like, and because the GPU is the smaller half of what self-hosting costs.

The honest 2026 position is that commercial APIs win on total cost for low and moderate volume, and self-hosting wins on cost only at high sustained volume with utilisation kept high enough to amortise both hardware and the people who run it. That is a narrower window than most business cases assume. It is also not the only reason to deploy privately: where data control is the driver, private deployment can be correct well below the cost crossover, and saying so plainly is more defensible than manufacturing a cost argument that will not survive twelve months of actuals.

This article separates the two arguments, because conflating them is what produces business cases that collapse at the first budget review.

The two cost equations

The comparison people usually make is incomplete on one side. Written out fully:

Self-hosted monthly cost equals GPU amortisation or rental, plus power, plus storage and network, plus operations labour, plus the recurring cost of evaluation, upgrades and incidents.

API monthly cost equals input tokens times the input rate, plus output tokens times the output rate, plus tooling and guardrails.

The API side is nearly complete as written, which is why it is easy to forecast. The self-hosted side has three terms that rarely appear in the spreadsheet and that between them usually decide the answer.

A published 2026 breakdown of a single A100 80GB deployment illustrates the proportions: $3,240 per month all in, split as $1,440 GPU rental, $1,500 DevOps labour and $300 infrastructure overhead. The compute is under half the total. At roughly 1 million tokens per day, that setup was cheaper than only the most expensive frontier APIs and still well above budget API pricing at the same volume.

The costs that erase the savings

• Operations labour. The line item above budgets 15 hours per month at $100 per hour purely for DevOps time on a modest deployment. That is a conservative figure for a system with real users, on-call expectations and a serving stack that changes underneath you.
• Infrastructure overhead. Networking, monitoring, logging, backups, security controls and orchestration. Budgeted at $300 per month in the same analysis, and that assumes you are reusing existing platform capability rather than building it.
• Evaluation and upgrade churn. Every model release, serving-stack upgrade and prompt migration carries benchmark runs, regression testing and redeployment. Sources fold this into engineering time rather than listing it separately, which is precisely why it goes missing from business cases. It is recurring, not one-off.
• Underutilisation. This is the quiet one. Fixed GPU cost is paid whether or not requests arrive. Bursty traffic, where a working-day peak sits above a near-idle night, means the effective cost per token can be several times the number your capacity calculation produced. APIs charge nothing when nobody is asking.

Underutilisation deserves its own analysis before anything is procured. Take a week of actual request logs, plot tokens per hour, and calculate the ratio of peak to mean. A ratio above three should make you cautious about purchase, because you will be sizing for peak and paying for it around the clock.

Where API pricing sits in 2026

API prices have fallen far enough to change the default answer. Reported mid-2026 ranges run roughly $0.50 to $3 per million input tokens and $1.50 to $15 per million output tokens. That spread is the crux of the decision, because which end you are replacing matters more than your volume does.

Against budget-tier APIs at the low end of those ranges, self-hosting is rarely justified on cost at any volume a mid-sized firm generates. Against premium frontier APIs at the high end, the crossover arrives considerably sooner and the case can be genuine.

So the first question is not “how many tokens do we send” but “which model tier does this workload actually require”. A classification or extraction workload that runs acceptably on a small open-weight model was never going to be expensive on an API either, and the self-hosting saving on it is correspondingly small. The workloads where self-hosting saves real money are those currently running on expensive models at steady volume.

The published break-even bands

The 2026 analyses do not converge on a single number, which is itself informative. They cluster:

• Under roughly 1 million tokens per day, APIs almost always win on total cost.
• 1 to 10 million tokens per day, it depends on model choice and utilisation. APIs usually still win unless you are replacing a frontier model or privacy requires otherwise. One optimistic premium-model comparison puts break-even as low as 5 to 10 million tokens per month on a single high-end GPU, but that figure assumes high utilisation and excludes most labour.
• 10 to 50 million tokens per day, self-hosting becomes plausible on cost where load is steady and operations are already mature.
• Above 50 million tokens per day, self-hosting often wins clearly, with reported advantages of 40% to 60% against premium APIs. One analysis of a Llama 70B-class workload at around 500 million tokens per day claims roughly a fivefold advantage.

Two other thresholds appear in the sources and are worth noting for scale: some private deployments reach cost parity within 3 to 6 months at around 50,000 queries per month, and several analyses put the enterprise-scale crossover at 100 million or more tokens per day. The variance between these claims is a function of what each author counted, which is the main reason to build your own model rather than adopt a published threshold.

When privacy is the reason, say so

Below roughly $20,000 per month of API spend, one 2026 analysis states the position bluntly: the API wins on every dimension except privacy. That is a useful framing because it removes the pressure to invent a cost story.

Private deployment is the correct answer regardless of the cost comparison when you need data residency and on-premises control, contractual isolation for regulated or client-confidential material, assurance that no third party retains prompts, outputs or embeddings, or a reduction in the legal and procurement exposure that comes with an external model provider. In regulated sectors, sensitive-IP work and internal copilots over confidential material, privacy is the deciding factor even where the API is cheaper.

A business case that says “this costs us $2,000 a month more than the API and here is the exposure it removes” is defensible in front of a board. A business case that claims cost savings which do not materialise costs you credibility on the next request.

What to do first

Instrument before you model. Collect a month of real token volumes split by input and output, by workload, and by hour of day. Most teams estimating this from memory are out by a factor that changes the answer.

Then run the comparison per workload rather than in aggregate. A firm typically has a small number of high-volume, low-complexity workloads that self-host well and a long tail of low-volume, high-complexity ones that do not. Hybrid is usually the correct architecture, with the sensitive and voluminous work running privately and the rest staying on an API. Forcing a single answer across the estate is what produces both overpriced private clusters and unacceptable data exposure.

Include a utilisation assumption you would defend in writing, and include the labour line. If the case only works at 80% GPU utilisation and your traffic is a working-day curve, the case does not work.

Finally, separate the decision from the timing. Rent capacity, run the workload privately for a quarter, and let the measured numbers decide whether to purchase. The cost of being wrong about rental is one quarter. The cost of being wrong about a hardware purchase is three years.