Hardening a self-hosted LLM stack

A self-hosted model does not make a deployment secure. It changes which risks you own. The data stops leaving your boundary, which is usually why the decision was made, and in exchange you inherit the full security surface of an inference service, a retrieval layer, a set of tools with production credentials, and a model artefact you downloaded from the internet.

The framing that holds up in review is that an LLM application is a privileged system. It reads untrusted text, makes decisions from it, and calls tools that touch real data. Nothing about the model layer justifies weakening the controls you would apply to any other service with that profile, and several of those controls need reworking because the trust boundaries sit in unfamiliar places.

The failures worth planning for are not exotic. Current 2026 guidance is consistent that the damage comes from over-privileged tools, untrusted retrieval, and inadequate runtime isolation rather than from clever prompts alone. That is a reassuring finding, because it means the work is mostly engineering discipline applied at five layers rather than research.

Treat every text input as untrusted

The starting assumption is that all external text is hostile: user prompts, retrieved documents, tool outputs, and conversation history. Retrieved content is the one teams miss, because it comes from an internal system and therefore feels trusted. A document uploaded by a supplier, an email ingested into a knowledge base, or a ticket written by a customer is attacker-controlled text sitting in your index waiting to be retrieved into a prompt.

The controls that work are structural rather than persuasive:

• Separate contexts explicitly. System, developer, tool, and user messages should be distinct, schema-constrained, and not concatenated into one undifferentiated block.
• Label retrieved passages as data. Retrieved content must not be able to override system instructions, and the prompt structure should make that separation explicit rather than implied.
• Gate tool calls outside the model. Tools exposed to an LLM are privileged interfaces. Authorisation belongs in a policy layer that evaluates the requested action against the caller’s entitlements, independent of what the model decided.
• Validate outputs before acting. Anything that becomes code, SQL, a shell command, a file write, or an HTTP request gets schema validation and allowlisting before execution, not after.
• Require approval for consequential actions. Payment release, data deletion, external communication, and permission changes belong behind a human step, with the model’s proposed action and its evidence shown to the approver.

The design test is simple: assume an attacker fully controls the model’s output for one request. If that gives them anything, the containment is in the wrong place.

Model supply chain and weights provenance

Open-weight models arrive as files from a public host, and most deployments trust them the way they would trust a base OS image, which is to say not carefully enough. Provenance verification is the control, and it is cheap.

Pin the exact weight artefact by digest rather than by tag or mutable name, so the thing you approved is the thing that loads. Verify checksums or signatures on every deploy, and specifically on redeploys and autoscaled instances, since those are the paths that skip the review the initial deployment received. Store weights on encrypted volumes and mount them read-only into the inference container, so a compromised serving process cannot modify what it is running.

Behind the mechanics, keep a model registry with provenance metadata: source, checksum, version, training and fine-tune lineage, approval date, and accountable owner. This is what makes the control auditable rather than merely present, and it is the artefact a regulator or an internal auditor will ask for.

Extend the same thinking to the retrieval corpus and the embedding pipeline. They are part of the supply chain, and current guidance is explicit that source validation and ingestion sandboxing belong there. A poisoned document in the index is a persistent injection vector with none of the visibility that a compromised model file would have.

Runtime isolation

Container and cluster hardening for model serving is ordinary practice applied consistently, and it is where the blast radius of everything above gets bounded.

• Run inference containers as non-root and drop all unnecessary Linux capabilities.
• Apply seccomp profiles and AppArmor or SELinux confinement where the platform supports it.
• Use read-only root filesystems, mounting only the minimum writable paths the runtime requires.
• Segment the network so the inference pod cannot reach training systems, vector stores, or admin services except through explicitly authorised paths.
• Enforce network policy plus mTLS for service-to-service communication, so a compromised component cannot move laterally on the assumption that internal traffic is trusted.
• Isolate sensitive workloads with separate namespaces, service accounts, and where the classification warrants it, separate node pools.

The segmentation point deserves emphasis. Inference services tend to accumulate network reach because they need retrieval, tools, logging, and metrics, and the default outcome is a pod that can talk to everything. Enumerate the destinations it genuinely needs and deny the rest, because that policy is what turns a successful injection into a contained incident.

Secrets and credentials

Secrets exposure is what converts a prompt injection into a broader compromise, and the controls are unglamorous.

Keep secrets out of the model context. API keys, database passwords, and signing keys should never be placed in a prompt, and where a token genuinely must be referenced, it should be short-lived and scoped to a single action. Use a vault or KMS-backed broker rather than environment variables for long-lived credentials, and prefer short-lived tokens for ingestion, tool access, and service authentication.

Tools should authenticate as dedicated service identities with narrowly scoped permissions, not as the human user and not as an admin service account that predates the project. The most damaging incidents in this space come from a tool with more authority than the task required, invoked by a model that was talked into it.

Add secret scanning to CI/CD and to the runtime log pipeline. Prompts and outputs pass through logs, and credentials that appear in either need to be caught quickly rather than discovered during a review.

Audit logging that survives an incident

Application request logging is insufficient for an LLM system, because the question after an incident is not which endpoint was called. It is what the model saw, what it did, and what the system allowed.

Capture the full inference lifecycle: who invoked the model, what context was supplied, which retrieval results were used, which tools were called with what arguments, what output was returned, and the auth context throughout. Record policy decisions as first-class events too: blocked tool calls, redacted prompts, denied retrievals, failed provenance checks, denied secret access. Those records are the evidence that your controls were operating, which is a different claim from the controls existing.

Store logs in an append-only or tamper-evident system where practical, and integrate them with your SOC rather than leaving them in an application log nobody watches. The same data supports anomaly detection: prompt volume spikes, unusual tool call sequences, GPU or cost abuse, and repeated jailbreak patterns from one identity. Retain according to your compliance regime, and redact secrets at write time so retention does not create a new exposure.

Where the OWASP list fits

The OWASP Top 10 for Large Language Model Applications remains a useful taxonomy and a reasonable checklist for coverage, and its examples still centre on the right thing: tools and plugins that process untrusted input without adequate access control, up to and including remote code execution. Use it to check that you have considered prompt injection, insecure output handling, data leakage, supply chain risk, and excessive agency.

Do not stop there. A checklist confirms you thought about a category; it does not tell you where the enforcement point belongs. The stronger position is to enforce policy outside the model and treat retrieval, tools, and execution as separate trust boundaries, each with its own authorisation.

What to do first

Start with the tool inventory, because it is where the real authority sits. List every tool the model can call, the credential it uses, and what an attacker gains from calling it with arbitrary arguments. Reduce the permissions until that answer is acceptable, and put the consequential ones behind approval.

Then add authentication and authorisation at the inference gateway with scoped claims and short expiry, pin and verify your weights, harden the serving containers, and turn on lifecycle audit logging. Those five are the practical control set, and they are all deliverable before a first production release. In our experience the teams that retrofit them do so after an internal review blocks the launch, at several times the cost of building them in.