Vector Store Index Corruption: Causes, Detection, and Recovery

Your RAG pipeline was working. Retrieval quality was good, stakeholders were satisfied, and the system was answering questions accurately. Then, gradually or suddenly, it stopped. Queries that used to return relevant context now return irrelevant fragments or nothing at all. The LLM starts hallucinating because it is working with bad context — or no context.

The embedding model has not changed. The documents are still there. The problem is in between: the vector store index is corrupted.

Index corruption is one of the most disorienting failures in production RAG because the symptoms mimic other problems. Bad retrieval results look like a chunking issue, an embedding issue, or a prompt engineering issue. Teams can spend days tuning the wrong component before discovering the index itself is broken.

This article covers the causes of vector store index corruption, how to detect it systematically, and how to recover without losing your entire index.

What Causes Index Corruption

Vector store indexes are complex data structures — HNSW graphs, IVF partitions, or flat indexes with metadata mappings. They are not simple key-value stores. Corruption happens when the internal consistency of these structures is violated.

Partial Writes During Indexing

The most common cause. When you add new vectors to an index, the operation involves multiple steps: inserting the vector, updating the graph structure (for HNSW) or partition assignments (for IVF), and writing metadata. If the process is interrupted mid-write — due to a crash, a deployment, a container restart, or a network timeout — the index can be left in an inconsistent state.

The result: some vectors are inserted but not connected to the graph, or metadata references point to vectors that do not exist. Searches return incomplete results because the graph traversal cannot reach the disconnected vectors.

Out-of-Memory During Indexing

Large batch indexing operations can exceed available memory, especially when the vector count approaches the limits of what the machine can hold in RAM. When the process is killed by the OOM killer, the index file on disk may be partially written.

This is particularly dangerous with memory-mapped indexes. The operating system may have flushed some dirty pages to disk but not others, leaving the index file in a state that no single point in time represents.

Version Mismatches

Vector store libraries evolve their on-disk formats between versions. Upgrading the library without migrating the index — or worse, having different services use different library versions against the same index — creates corruption that manifests as silent retrieval degradation rather than hard errors.

A common scenario: a development environment is updated to the latest library version while production remains on the previous version. An index built or modified in development is deployed to production. The older library reads the newer format partially, missing vectors or misinterpreting graph edges.

Concurrent Write Conflicts

Multiple processes writing to the same index simultaneously without proper locking is a recipe for corruption. This happens more often than expected — a re-indexing job starts while the application is still serving writes, or two worker processes both attempt to add vectors from different document batches.

Hardware-Level Failures

Disk corruption, failing SSDs, and unreliable network-attached storage can flip bits in the index file. These failures are rare but produce the most confusing symptoms because the corruption is random and does not follow any logical pattern.

How to Detect Index Corruption

Detection is the hard part. Corruption rarely announces itself with a clear error message. Instead, you see degraded retrieval quality that could have many causes.

The Diagnostic Framework

Step 1: Establish a retrieval baseline. Maintain a set of test queries with known-good expected results. Run these queries periodically (daily or after every indexing operation). If the results diverge from expected, something has changed — and if the documents and embeddings have not changed, the index is the prime suspect.

Step 2: Check vector count consistency. Compare the number of vectors in the index with the number of chunks your pipeline produced. If they diverge, vectors were either lost during indexing or the index dropped them due to corruption. This is the single most reliable corruption signal.

Step 3: Run a nearest-neighbor sanity check. Query the index with a vector that is already in the index (use the exact embedding of a known chunk). The top result should be that exact chunk with a distance of zero (or a similarity score of 1.0). If it is not, the index structure is broken — the graph cannot reach vectors it should contain.

Step 4: Check for metadata orphans. Query for documents by metadata filter and compare results with what should exist. Missing results indicate that either the vectors or their metadata mappings are corrupted.

Step 5: Validate index file integrity. If your vector store supports it, run the built-in index validation or repair command. FAISS, for example, does not have a built-in validator, but Qdrant and Milvus expose health-check endpoints that report index status.

Symptoms That Point to Corruption vs. Other Issues

Symptom	Likely Corruption	Likely Other Cause
Retrieval returns zero results for queries that previously worked	Yes — vectors unreachable	Embedding model changed
Retrieval returns results but they are irrelevant	Possible — partial graph corruption	Chunking strategy or prompt issue
Vector count lower than expected	Yes — vectors lost during write	Indexing pipeline bug
Queries work for recent documents but not older ones	Yes — older index segments corrupted	Re-indexing overwrote old data
Intermittent failures (sometimes works, sometimes does not)	Yes — partial graph damage	Network or resource contention
All queries return the same results regardless of input	Yes — index structure collapsed	Embedding model producing identical vectors

The Recovery Decision Tree

When you have confirmed or strongly suspect index corruption, follow this decision tree to choose the right recovery strategy.

1. Can you rebuild the index from source documents?

If yes, this is always the safest option. Re-run your entire indexing pipeline from the original documents. This eliminates any corruption and produces a known-good index. The cost is compute time and temporary service degradation during re-indexing.

If rebuilding from source is feasible, do it. Every other recovery strategy is a compromise.

2. Do you have a recent backup of the index?

If yes, restore from the backup and then re-index only the documents added since the backup timestamp. This is faster than a full rebuild and produces a reliable result, assuming the backup itself is not corrupted.

Verify the backup before restoring: check vector count, run the nearest-neighbor sanity check, and validate metadata integrity on the backup before promoting it to production.

3. Does your vector store support index repair?

Some vector stores (Qdrant, Weaviate) support index compaction or repair operations that can fix certain types of corruption — particularly orphaned vectors and inconsistent metadata. These operations are not guaranteed to fix all corruption types, but they are worth attempting before a full rebuild.

4. Can you identify and re-index only the corrupted segment?

If your indexing pipeline tracks which documents were being processed when the corruption occurred (for example, the batch that was running when the OOM kill happened), you can attempt to re-index only that batch. Delete the vectors associated with that batch and re-insert them.

This is a targeted repair that preserves the rest of the index. It works well for partial-write corruption but will not fix structural damage to the graph.

5. None of the above?

Full rebuild from source is your only reliable option. This is why maintaining source documents and a reproducible indexing pipeline is non-negotiable for production RAG systems.

Prevention Checklist

Corruption recovery is expensive and disruptive. Prevention is cheaper. Implement these practices before your first corruption event.

Atomic writes: Use vector store write operations that are transactional or can be rolled back. If your store does not support transactions, implement write-ahead logging at the application level.
Memory headroom: Configure indexing batch sizes to use no more than 60-70% of available memory. Monitor memory usage during indexing and reduce batch sizes proactively rather than waiting for OOM kills.
Write locking: Ensure only one process writes to the index at a time. Use distributed locks if multiple services need write access.
Version pinning: Pin your vector store library version across all environments. Include the library version in your index metadata so you can detect mismatches.
Regular backups: Back up the index after every successful indexing operation. Automate backup validation (vector count check, nearest-neighbor sanity check).
Baseline test queries: Maintain a suite of test queries with expected results. Run them after every indexing operation and alert on divergence.
Monitoring: Track vector count over time, query latency percentiles, and retrieval quality metrics. Sudden changes in any of these signal potential corruption.
Graceful shutdown: Ensure your indexing process handles SIGTERM gracefully — flush pending writes and close the index cleanly before exiting.

Where Ertas Fits

Ertas Data Suite builds RAG indexing pipelines as observable, node-based workflows on a visual canvas. The Vector Store Writer node tracks exactly which chunks were written, when, and from which source documents. If an indexing operation fails partway through, you can see precisely which documents were processed and which were not — no guessing, no log file archaeology.

The pipeline audit trail means recovery is targeted rather than brute-force. Instead of rebuilding the entire index because you are not sure which documents were affected, you re-run the pipeline for only the documents that were in-flight when the failure occurred.

For teams operating RAG systems in production, the cost of index corruption is not just the recovery time. It is the period of degraded retrieval quality before anyone notices the problem. Observable pipelines with built-in consistency checks are the difference between catching corruption in minutes and discovering it weeks later when a stakeholder reports that the system "seems worse lately."

Key Takeaways

Vector store index corruption is a when-not-if problem for production RAG systems. The most common causes — partial writes, OOM kills, version mismatches — are all preventable with proper operational practices. Detection requires proactive monitoring rather than reactive investigation. And recovery is dramatically easier when you maintain source documents, regular backups, and a reproducible indexing pipeline.

Build your RAG infrastructure with the assumption that the index will need to be rebuilt. The teams that recover fastest are the ones who planned for it.