A private model deployment usually gets built with more care than the thing that watches it. The inference stack is sized, hardened and network-isolated, and then quality is assessed by someone trying a few prompts and declaring the output good. Six weeks later a prompt is edited, a retrieval index is rebuilt, or the serving stack is upgraded, and nobody can say whether the system got better or worse, because there was never a measurement to compare against.
This matters more on private infrastructure than in a cloud deployment, for a reason that is easy to miss. Teams using a commercial API inherit the provider’s regression testing on the model itself; the model behaves consistently because someone else is holding it steady. Teams running open-weight models on their own hardware own every variable: the weights, the quantisation, the serving engine, the context window handling, the retrieval layer and the prompts. Any of those can change under you, and several will.
The constraint is that the obvious tooling for this problem is cloud-first. Most evaluation and tracing platforms are built around a SaaS backend that receives your prompts, completions and retrieved context, which is precisely the data that motivated the private deployment. Solving that is a tooling selection problem before it is an engineering one.
What the private-infrastructure requirement actually adds
Five properties separate tooling that works inside a controlled perimeter from tooling that merely claims enterprise readiness.
- Fully local execution. Evaluations, trace storage, dashboards and scoring all run without contacting an external backend. This has to include the scoring path, because an evaluation framework that calls a hosted judge model is exporting exactly the content you were protecting.
- Prompt regression testing. A fixed test set that runs on every change to prompts, retrieval indexes or model versions, producing a comparable score rather than an impression.
- Trace capture with useful payloads. Prompts, completions, retrieved context, tool calls, token counts, latencies and costs, per request. Span structure alone is not enough; the payload has to be rich enough to reconstruct what happened.
- Quality drift monitoring. Output quality tracked over time through automated evaluation pipelines and human feedback, tied to defined rollback thresholds.
- Genuine private deployment modes. Private VPC, on-premises or air-gapped operation with no internet dependency, including for licence checks and telemetry.
That last point catches people. A platform that runs on your hardware but phones home for licence validation fails an air-gap requirement, and it usually fails it at the worst possible moment.
The tooling landscape in 2026
The 2026 comparisons sort the field by deployment model more usefully than by feature list.
Open-source core with a strong self-hosting story: Langfuse, Arize Phoenix and Comet Opik all appear consistently among self-hostable platforms. These are the default starting point when you want the code inside your perimeter and no vendor relationship in the critical path.
Commercial with explicit private deployment: Braintrust is described as self-hosted and on-premises capable including private VPC and air-gapped use, which makes it the more direct answer when the requirement is specifically a supported air-gapped evaluation platform. Orq.ai offers cloud, VPC and on-premises deployment across one product line, which suits organisations that want a single vendor across environments of differing sensitivity.
Cloud-first with limited on-premises options: LangSmith and Maxim are both characterised in 2026 comparisons as cloud-first with self-hosting either restricted or enterprise-tier only. They are capable products and they are the wrong shape for a fully offline requirement.
The selection rule is short. If you want an open-source core and are comfortable operating it, start with Langfuse, Phoenix or Opik. If you need a supported product with an explicit air-gap story, look at Braintrust. If you need one vendor spanning cloud, VPC and on-premises, look at Orq.ai. Do not build a fully offline requirement on a cloud-first platform and expect to negotiate your way to it later.
The architecture that holds up
Four layers, and the discipline is in how they connect rather than in any one of them.
The eval harness maintains a curated test set of real questions with reviewed answers, and runs on every change to prompts, model versions or retrieval indexes. The word doing the work is curated. A set assembled from real user questions and real failures is worth several times a set generated by a model, because generated questions test the distribution the generator imagines rather than the one your users produce.
The tracing layer instruments every span in the pipeline: the inbound request, retrieval, the model call and each tool invocation. Align on OpenTelemetry-shaped instrumentation, because 2026 observability tooling is built around trace data in that shape and it keeps you portable between backends. The operational requirement goes beyond standard spans, though. The payloads need prompts, completions, retrieval context, token counts, latencies and costs, otherwise you have timing data for a system whose failures are almost never about timing.
The metrics layer tracks latency, token usage, queue depth, throughput and cost alongside the quality measures. On private infrastructure this layer earns its keep quickly, because queue depth and throughput are where capacity problems announce themselves before users notice them.
Quality gates and a model registry. Require evaluation thresholds to pass before a version is promoted, and roll back when quality falls below baseline. Version model artefacts with their metadata: training-data hash where applicable, hyperparameters, quantisation settings and evaluation scores. On a private stack you will at some point need to answer which exact artefact produced a given output, and a registry is the difference between a five-minute answer and a forensic exercise.
The connection that matters most is between the tracing layer and the eval harness. Production traces and offline evaluations should share the same telemetry shape, so that a production failure can be lifted directly into the regression set without being rewritten by hand. Teams that skip this end up with two disconnected views of quality and spend their time reconciling them.
How this runs in practice
Before deployment, run the offline benchmark suite against a fixed prompt set and compare candidate models or prompt versions on identical inputs. This is the only comparison that means anything when you are choosing between open-weight models, because published benchmark scores tell you about general capability rather than about your documents and your questions.
During rollout, route a small fraction of traffic to the new version and compare quality, latency and cost. On private infrastructure the cost dimension behaves differently from an API deployment: a version that consumes more tokens per request consumes more of a fixed capacity, so throughput regressions show up as capacity problems rather than as a larger invoice.
In production, compare live output distributions and evaluator scores against baseline continuously. Drift on a self-hosted stack rarely comes from the model, which is frozen unless you change it. It comes from the inputs and the retrieval corpus moving, which is a slower and less obvious failure than a model update and is missed by teams watching only for deployment events.
For sensitive workloads, keep prompts, completions, traces, datasets and evaluation results inside the same perimeter as the model. Evaluation data is often more concentrated than production traffic, because a test set is a curated collection of your hardest and most representative cases. It deserves the same handling as the source documents.
What to do first
Build the regression set before building anything else. Fifty real questions with reviewed answers, collected from the people who will use the system, is a week of work and it is what makes every later decision measurable. Without it, model selection, prompt changes and retrieval tuning are all conducted on opinion.
Then instrument for full payload capture from the first day of the pilot, not after launch. Traces you did not collect cannot be recovered, and the pilot period is when the interesting failures happen and when you still have the attention to look at them.
Choose the platform against your hardest deployment constraint rather than your typical one. If any environment in scope is air-gapped, select for that environment, because running two evaluation platforms to cover two environments is a cost you will pay every week thereafter.
FAQ
Can we use a cloud eval platform if only the scores leave our network, not the prompts?
Sometimes, but check what the SDK actually transmits rather than what the marketing page says. Most tracing clients send the full span payload including prompt and completion text by default, and the redaction hooks are opt-in and easy to bypass with a new instrumentation call. If your requirement is that no prompt content leaves the perimeter, a self-hosted deployment enforces that structurally, whereas a cloud platform with redaction enabled relies on every future integration remembering to configure it.
How large does the regression test set need to be?
Smaller than teams expect, and curated rather than sampled. Fifty to two hundred real questions with reviewed reference answers, chosen to cover the failure modes you actually care about, will catch more regressions than a thousand randomly sampled queries. Add a case every time production produces a wrong answer, which turns incidents into permanent coverage. The set only has value if it runs on every change to prompts, model versions and retrieval indexes, so keep it fast enough that nobody is tempted to skip it.
What do we monitor once the system is live?
Two layers. Operational metrics come first: latency, token usage, queue depth, throughput and cost per request, because most user-visible failures are operational rather than qualitative. Then quality signals: automated evaluator scores on sampled production traffic compared against your baseline distribution, retrieval hit rates, refusal and fallback rates, and human feedback where you can collect it. Pair the quality signals with an explicit rollback threshold agreed before deployment, otherwise drift gets debated instead of acted on.