Sizing on-prem LLM inference hardware

Hardware sizing for on-premises inference is where private AI programmes most often go wrong in a way that is expensive to correct. The failure is rarely buying too little. It is buying a configuration chosen from a benchmark headline rather than from the workload, then discovering that throughput collapses at production context lengths, or that a card with impressive compute numbers is starved by memory bandwidth.

The sizing logic is simpler than the GPU market makes it look. First, the weights have to fit in VRAM. Then you reserve additional VRAM for KV cache, activations, and runtime overhead. A working estimate is parameters multiplied by bytes per parameter, plus 30 to 50 percent for serving. INT4 is roughly 0.5 bytes per parameter, FP8 roughly 1 byte, FP16 roughly 2. Everything else in the sizing conversation is a refinement of that arithmetic.

The second principle matters just as much and gets less attention: the best card is not the one with the highest compute figure but the one with enough memory bandwidth and enough VRAM to hold your model and its KV cache under your concurrency. Inference serving is bandwidth-bound far more often than it is compute-bound.

The VRAM arithmetic

Apply the formula and the picture becomes concrete quickly.

Model size	FP8 weights	INT4 weights	Practical VRAM with serving overhead
7B	~7 GB	~3.5 GB	5–10 GB for light serving, more at long context
13B	~13 GB	~6.5 GB	9–20 GB
34B	~34 GB	~17 GB	25–50 GB
70B	~70 GB	~35 GB	45–100+ GB

Two adjustments apply in practice. Quantization methods carry metadata overhead of around 10 percent depending on the approach, so treat the weight figures as floors rather than targets. And KV cache grows with both context length and batch size, which is why a configuration that benchmarks cleanly at 4K context can fall apart at 64K with a dozen concurrent users. If your use case is long-document retrieval, size the KV cache deliberately instead of letting it absorb whatever headroom is left.

Choosing a GPU tier

The market splits into four bands, and most enterprises need only two of them.

• H200 and B200. The frontier tier: large models, high throughput, multi-GPU serving with NVLink. Reported figures give 8-GPU H200 systems around 12,400 tokens per second on Llama 4 Scout and 2,864 tokens per second on DeepSeek V3 at FP8, and one April 2026 comparison put B200 throughput at up to 4.9 times an RTX PRO 6000. This tier is justified by concurrency and model size, not by prestige.
• L40S, RTX 6000 Ada, and RTX PRO 6000 Blackwell. The serious single-node production tier. L40S and RTX 6000 Ada are 48 GB cards with strong bandwidth and no NVLink, which caps how well they scale across a box but makes them a good fit for 34B-class serving and quantized larger models. RTX PRO 6000 Blackwell is now a standard single-GPU production choice. For a large share of internal enterprise workloads this band is the right answer and the frontier tier is over-buying.
• AMD MI325X and the wider MI300 family. The large-memory alternative when you want AMD compute economics and can accept ROCm’s software maturity relative to CUDA. The trade is real: vLLM, TensorRT-LLM, and FlashAttention are strongest on NVIDIA, so factor in engineering time for tuning and a narrower set of tested paths.
• Consumer cards, RTX 4090 and 5090. Genuinely useful for development environments, small-team deployments, and quantized models. Not a shared production tier, because capacity and bandwidth put a hard ceiling on concurrency well before compute does.

Quantization is the main lever

Quantization does more for your hardware budget than any other single decision. Moving a 70B model from FP16 to INT4 takes the weights from roughly 140 GB to roughly 35 GB, which is the difference between a multi-GPU node and a single 80 GB card with room for KV cache. That is not a marginal efficiency gain, it changes which purchase order you write.

The trade-off has to be measured rather than assumed. FP8 usually preserves quality well and runs fast on hardware with native support, at double INT4’s memory footprint. INT4 is where you gain the most room and where quality loss becomes task-dependent: retrieval-grounded answering, extraction, and classification generally tolerate it well, while multi-step reasoning and code generation are more sensitive. Build a small evaluation set from your real workload, run it at each precision, and let the results decide. This is a half-day of work that regularly saves six figures of hardware.

Estimating throughput before you buy

Throughput is driven by memory bandwidth, batching, context length, and whether the model fits entirely in fast memory. Three heuristics cover most planning.

• If the model fits comfortably in VRAM and context is modest, expect throughput near published figures for the card class.
• If you are near the memory limit, throughput degrades sharply because KV cache pressure forces smaller batches or offloading, and the degradation is non-linear.
• If you need high concurrency, prioritise HBM-equipped datacentre parts. Capacity and bandwidth dominate serving performance once several users share a card.

The number to plan against is not peak tokens per second. It is tokens per second at your p95 context length and your expected concurrency, which is usually a fraction of the headline. Ask vendors for figures at your context length, and if you cannot get them, run the benchmark yourself on rented capacity before committing to hardware.

Matching tier to workload

Workload	Practical hardware tier
Prototyping, edge, small assistants	CPU-only or a low-end GPU, small models and low throughput only
7B–13B internal tools	16–24 GB GPUs, especially with quantization
34B-class serving	48 GB cards such as L40S or RTX 6000 Ada as the comfortable lower bound
70B-class single-node serving	80 GB+ datacentre GPUs, or aggressive quantization on high-memory cards
Frontier and multi-user production	H200 or B200 class, usually multi-GPU with NVLink

CPU inference deserves its place in this table rather than dismissal. Ollama and LM Studio support CPU-only execution, and models such as Qwen3 0.6B or Llama 3.2 1B run on a laptop with 8 GB of RAM. At roughly 2 to 10 tokens per second it is unusable for interactive multi-user work and perfectly adequate for overnight batch classification of documents, which is a larger share of real enterprise AI work than the interactive-assistant framing suggests.

Where to start

Size from measured demand, not from the largest model you might one day want to run. Take your intended workload, establish the p95 context length and peak concurrency, pick the smallest model that passes your quality bar on your own evaluation set, then apply the VRAM formula with a 40 percent serving margin. That gives you a card class. Validate it on rented GPU capacity for a week before purchase, because a week of cloud GPU time is cheap against a mis-specified cluster.

Two practical notes on the buy itself. Budget for the surrounding infrastructure, since GPU nodes need networking and storage paths that match, and an underfed card is a wasted card. And leave headroom for model turnover: open-weight releases move faster than depreciation schedules, and the configuration you buy will serve a different model within a year. Cards with more memory than today’s model needs age considerably better than cards chosen to fit it exactly.