Versioning embeddings and reindexing without downtime

Changing the embedding model invalidates every vector you have stored, because distances between vectors from two different models mean nothing. A query embedded with the new model ranks old vectors by noise. So you re-embed the whole corpus into a versioned index, keep the old one serving live traffic, and cut over only once an eval set says the new one retrieves at least as well.

The trap is treating embeddings as durable data. They are a derived artifact, like a compiled binary. The source of truth is your chunks and their metadata; the vectors are a function of those chunks and a specific model at a specific version. When the function changes, the output is stale, all of it, at once. Teams that store embeddings without recording which model produced them find this out the hard way the first time they upgrade, usually when retrieval quality falls off a cliff and nothing in the logs explains why.

Why an upgrade forces a full re-embed

You cannot backfill. There is no clever migration that re-embeds the documents that changed and leaves the rest alone. A vector from text-embedding-3-large and a vector from a newer model do not live in the same space, are frequently not even the same dimensionality, and their cosine distances sit on scales you cannot compare. Mixing them in one index means a query will confidently rank passages by an artifact of which model happened to embed them. That is the retrieval equivalent of comparing prices in two currencies with no exchange rate.

So every model change forces a decision you should make deliberately rather than stumble into:

• The embedding model itself changing. A vendor deprecates a version, or you move from a hosted model to one you run in-house to cut third-party dependency, the kind of ICT concentration risk DORA expects a financial entity to manage. Full re-embed.
• The chunking strategy changing. Different boundaries or window sizes mean different input text, so different vectors. Full re-embed, and usually a re-eval of retrieval quality, because chunking moves quality more than the model does.
• Preprocessing changing. You start stripping boilerplate, normalizing tables, or resolving entity names before embedding. The input text changed, so the vectors are stale.

Notice that two of these three have nothing to do with the model. Anything that alters the text handed to the embedder invalidates the output. This is why you treat the embedding step as a pure, versioned function and record its full signature, not just the model name.

Give every vector a version key

Every stored vector needs to carry enough lineage to answer one question: exactly what produced you? In practice that is a compact version key attached to each vector and to the index as a whole. We record four things:

• The embedding model and its exact version string, never a floating alias a vendor can silently repoint.
• The chunking strategy and its parameters: boundary rules, target token length, overlap.
• The preprocessing pipeline version, as a hash or tag of the normalization and entity-resolution code.
• The dimensionality and the distance metric, so a mismatch fails loudly at query time instead of returning garbage.

Bundle those into a single embedding_version identifier and stamp it on the index. Now the query path can assert that the vector it produced for the query and the index it is searching share the same version. If they disagree, the query errors rather than returning plausible nonsense. That assertion earns its keep in finance. A wrong passage retrieved because the query and index versions drifted apart is the confident-sentence-with-a-wrong-number failure: invisible in a demo, expensive in an audit.

Keeping this lineage next to the chunk metadata, rather than in a separate registry you have to reconcile, means the answer to “which model embedded the passage behind this answer” is a lookup, not an investigation. When someone asks you to reconstruct why a retrieval-augmented answer came out the way it did six months ago, that lineage is your audit trail.

The cutover: build alongside, prove, then swap

Reindexing without downtime is a blue-green deployment applied to vectors. You never mutate the live index in place. You build a new one beside it and redirect traffic only once it has earned the switch.

The sequence we run:

1. Build the new index cold. Re-embed the whole corpus into a fresh index or namespace tagged with the new embedding_version. This runs in the background against a snapshot of the chunks and never touches the index serving production. It can take hours, which is fine, because nobody is waiting on it.
2. Freeze and reconcile the delta. Documents ingested while the backfill ran need embedding too. Track ingestion timestamps and embed that delta into the new index before cutover, so it is not missing the last few hours of filings.
3. Evaluate before anyone trusts it. Run a fixed eval set of real queries with known-correct passages against both the old and new index. Compare recall at your production scoping and reranked precision. If the new model does not win or tie, you stop here and have lost nothing, because the old index never stopped serving. Never cut over on a vendor benchmark; measure on your corpus.
4. Flip the pointer atomically. Cutover is a single change to which index the query path reads from, a config flag or an alias swap. One request reads the old index; the next reads the new one. No window exists where a query sees a mixture.
5. Keep the old index warm for rollback. Hold it for a defined period. If reranked precision drifts or latency regresses in production, you flip the pointer back in seconds. Only after the new index has held up do you reclaim the old one’s memory, which for HNSW is the expensive resource.

For a large corpus you can shadow-read before the flip. Send a sample of live queries to both indexes, log both result sets, and diff them offline. That catches regressions the eval set missed, because real query traffic is always stranger than your eval set. Watch the exact-token identifier queries in particular, the CUSIPs, LEIs, and section references where a dense-model change can quietly shift behavior.

Run this often enough and reindexing stops being the event you dread at quarter-end. It becomes routine maintenance, and the reason it stays safe is boring. Vectors are disposable. The chunks and their lineage are the source of truth. And the query path refuses to search any index whose version it cannot verify.