RAG on private infrastructure

Most private retrieval projects stall in the same place. The proof of concept answered questions convincingly against a hundred documents, and the production version, pointed at a real document estate with real access rules, returns fragments that are close to the topic but useless as evidence. Nothing in the model changed. What changed is that the retrieval layer is now doing work the demo never asked of it.

Retrieval-augmented generation on your own infrastructure is a data engineering problem wearing an AI costume. The generation step is the part that is nearly solved: a self-hosted open-weight model with a decent context window will summarise correctly supplied evidence reliably. The parts that decide whether the system is usable are what you indexed, how you split it, whether the right passage can be found by the words people actually type, and whether a user who should not see a document is structurally prevented from seeing it through the answer.

That framing matters for regulated deployments, because the same properties that make a retrieval system accurate also make it defensible. A pipeline that stores provenance for every chunk can cite its sources. A pipeline that filters candidates by entitlement before ranking can answer an auditor’s question about who could have seen what. These are design decisions taken at ingestion time and expensive to retrofit.

Choosing a store, and not over-choosing

The self-hosted vector store market has consolidated around a handful of defensible answers, and the differences matter less than the internet suggests.

• pgvector is the right default when Postgres is already central and the corpus is modest. Vectors, metadata, and relational joins live in one database with one backup story and one permission model. Fewer moving parts is a real operational advantage in a regulated estate.
• Qdrant is the common production default for self-hosted retrieval where filtered search latency matters. It deploys cleanly under Docker or Kubernetes, handles metadata filtering well, and does not require a cluster to be useful.
• Weaviate earns its place when native hybrid search and schema modelling are central to the application, and it carries strong multi-tenancy positioning.
• Milvus is for genuinely large scale and heavy continuous ingestion, where separating ingest from query paths is the point. It costs more to operate; choose it when the scale justifies that.

The failure mode we see most often is not picking the wrong engine. It is picking one early, building everything against its query API, and discovering at migration time that retrieval logic, chunking rules, and ACL filtering were all entangled in vendor-specific calls. Keep an interface between your retrieval logic and the store. It is twenty lines of indirection that makes the engine decision reversible.

The private embedding layer

A retrieval stack on private infrastructure needs a private embedding service, and this is where teams accidentally reintroduce the dependency they were trying to remove. Every document you embed through a hosted API is a document you sent outside the boundary, and that includes the re-embedding runs nobody plans for.

The current pattern is Hugging Face’s text-embeddings-inference server, running an open encoder such as BGE-M3 on your own GPUs or, for smaller corpora, on CPU. TEI is worth the specific mention because it gives you token-based dynamic batching, optimised transformer kernels, OpenTelemetry tracing, and Prometheus metrics. In practice this means embedding throughput you can measure and tune rather than guess at, which matters when a full re-index is a scheduled operational event rather than a one-off.

Add a self-hosted cross-encoder reranker on top of first-stage retrieval. Dense top-k retrieval alone routinely leaves the passage you need just below the cut line, and reranking the top 50 down to the top 5 recovers those cases at a cost measured in tens of milliseconds. It is the single highest-return component most private stacks are missing.

One planning point: your embedding model choice is a commitment. Changing it means re-embedding the entire corpus, so treat the switch as a migration with a backfill window, and keep the raw text and chunk boundaries stored separately from the vectors so a re-embed does not mean a re-parse.

Chunking decides your ceiling

Chunking is the most common failure point in production retrieval, and fixed-size token windows are the usual cause. A 512-token splitter cuts through the middle of a procedure, separates a table from its header row, splits a contract clause from its definitions, and severs a question from its answer. Retrieval then returns a fragment that is topically correct and evidentially worthless, and the model fills the gap with something plausible.

The correction is document-aware splitting rather than one universal splitter:

• Route by document type. Policies, manuals, FAQs, contracts, code, and HTML pages each have structural units worth preserving, and the parser should know which it is handling.
• Keep structural units intact: headings with their bodies, list items, table rows with their headers, code blocks, and numbered legal clauses.
• Use overlap sparingly. Overlap patches boundary damage; it does not prevent semantic breakage, and it inflates the index.
• Store provenance on every chunk: document ID, section heading, page number, source system, ACL tags, and ingestion timestamp. This is what makes citations, filtering, and debugging possible later.

Chunking rules are worth more engineering attention than model selection. We have seen recall improve more from a document-type-aware splitter than from any model upgrade in the same project.

Hybrid retrieval and permission filtering

Pure vector search misses exact tokens: part numbers, error codes, policy clause references, ticket IDs, and rare proper nouns. These are precisely the terms that appear in real enterprise queries. Hybrid retrieval combines a dense channel for semantic and paraphrase matching with a sparse BM25 channel for exact terms, then fuses or reranks the two candidate sets before generation. For corpora of policies, tickets, logs, or technical documentation, this is not an optimisation. Dense-only retrieval will fail on the queries your users care most about.

Permission-aware retrieval is the other non-negotiable. Attach ACL metadata to each chunk at ingestion: tenant, department, role, classification, and source system. Filter by entitlement before ranking, so disallowed chunks never reach the candidate set. Where isolation matters more than cross-tenant search, use per-tenant collections or namespaces instead of relying on filters alone. And enforce the same entitlement check outside the vector store, because a highly relevant chunk that was not permission-filtered is a data leak with good relevance scores.

Evaluation built from real failures

Synthetic evaluation sets are generated from the corpus you already indexed, which means they test the retrieval you already have and hide the two failure modes that matter: chunk-boundary damage and exact-match misses. Build the evaluation set from real user queries instead, weighted towards the ones that failed.

Track a small set of numbers per release: retrieval recall at k measured against a labelled gold chunk, MRR or nDCG for ranking quality, answer faithfulness and citation correctness, access-control correctness on permission-sensitive queries, and stage-level latency for embed, retrieve, rerank, and generate. Stage-level latency is worth the instrumentation, because “the assistant is slow” usually resolves to one stage and the fix is different for each.

Where to start

Index one document set with real access rules rather than a broad corpus with none. Access control is the constraint that reshapes the architecture, and discovering it late means rebuilding ingestion.

Write the evaluation set before the retrieval code, from real questions with their correct source passage identified. Fifty labelled cases are enough to make every subsequent decision empirical.

Then build the pipeline in the order that failures occur: parsing and chunking first, hybrid retrieval second, reranking third, generation last. In our experience the generation step is rarely where the project is lost, and teams that tune prompts before fixing chunking spend months treating a symptom.