Incremental RAG indexing without losing the old index

Treat indexing as a state transition

An indexing job is not just a loop that reads files and writes vectors. It changes the version of knowledge that retrieval is allowed to use. That makes the publish boundary more important than the loop itself.

If new chunks replace old chunks one at a time, users can query a half-updated collection. A document may have new text with old metadata, or only some of its chunks may exist. The system is technically online, but the answers are difficult to trust.

A safer model separates preparation from publication. Build the next document version away from the active one, validate it, then make one small state change that tells retrieval which version is current.

Give each source a durable identity

File names are weak identifiers. They change, collide, and say nothing about whether the content is actually different. Keep a small indexing record for each source so the job can make a deliberate decision before it calls an expensive parser or embedding model.

Source key. A stable identifier from the business system, storage object, or document database.
Content hash. A hash of the bytes or normalized text. Timestamps can be kept as hints, not proof of change.
Pipeline version. The parser, chunking rules, embedding model, and index schema that produced the active version.
Lifecycle state. Discovered, processing, staged, active, failed, or deleted. A job should never have to guess where it stopped.

source_key
content_hash
pipeline_version
active_version
staged_version
status
last_error
updated_at

Minimal indexing record

Use a publish boundary

The exact storage model can vary. The useful property is the same: retrieval sees either the old complete document or the new complete document, never a mixture.

Discover. Read source metadata and compare it with the indexing record. Mark missing sources as deletion candidates, but do not delete them yet.
Prepare. Parse, normalize, split, and enrich the changed document. Keep the resulting chunks under a new version identifier.
Embed and stage. Write the full set of vectors and metadata without changing the active version.
Validate. Check that the staged version has the expected document identity, non-empty text, valid vectors, and required retrieval metadata.
Publish. Switch the active version in one durable operation. If the storage engine cannot do this atomically, use an alias or a separate manifest that retrieval reads first.
Clean up. Remove superseded vectors later. Cleanup should not sit on the critical path of a successful publish.

Degraded mode is a product decision

Degraded mode should be decided before an outage. Otherwise every dependency failure turns into an improvised choice between returning an error and returning an answer based on incomplete data.

The safest fallback is often the last known-good index. It may be stale, but it has a clear boundary and can be labelled as stale. A partially written new index has neither property.

Examples of explicit failure behaviour
Failure	Ingestion behaviour	Retrieval behaviour
Embedding service unavailable	Keep the source pending and retry with limits.	Serve the active index. Do not publish empty vectors.
Parser rejects one document	Quarantine that source and retain its old active version.	Continue serving other documents and expose the failed source to operators.
Vector store is slow	Pause new batches when a bounded timeout is reached.	Return a clear temporary failure, or use a tested secondary search path if one exists.
Metadata store unavailable	Do not start jobs that cannot record their state.	Keep reading the active index only if its ownership and version remain unambiguous.

Retry the operation, not the ambiguity

Retries are useful for temporary network and service failures. They are dangerous when the write may already have succeeded and the operation is not idempotent. Give each job, document version, and chunk a deterministic key so the same work can be attempted again without producing duplicates.

A malformed file is not a temporary failure. After a small number of attempts, move it out of the normal queue and record enough context to repair it. One bad source should not block every healthy source behind it.

Put limits and jitter on retries. Infinite retry loops hide incidents and consume workers.
Keep failed work visible. A dead-letter queue without an owner is only another place to lose documents.
Make cancellation safe. Staged data can remain for diagnosis, but it must not become active by accident.

Log the document journey

An ingestion dashboard does not need dozens of charts. It needs to answer a few operational questions quickly: what changed, what is active, what failed, and whether retrieval is using stale data.

Correlate source key, job ID, pipeline version, and published document version.
Count discovered, skipped, staged, published, quarantined, and deleted sources separately.
Record dependency latency and failures without logging document text or other sensitive content.
Expose index freshness and the last successful publish time to operators.

Before the next indexing release

Can a document be reprocessed without creating duplicate active chunks?
Can retrieval identify one complete active version?
Does a failed new version leave the previous version usable?
Are deletion and cleanup delayed until publication is safe?
Can an operator find one failed source without reading application logs line by line?
Does the UI say when results may be stale?