Fix: Incarnation-scope SQLite vector store collection resources - #1537

Purpose of the change

Both SQLite-backed vector stores let a collection handle reach the collection that replaces it. Native resource names derive from (namespace, name) alone, so deleting a collection and creating one with the same name rebuilds resources under the same names, and a handle opened before the deletion keeps addressing them.

Measured before this change, single-process, no concurrency involved:

SQLiteVectorStore is the worse case. SQLiteVectorStoreCollection captures index_path and search_engine at open time and _maybe_save_index writes that engine to that path, so a handle to a deleted collection rewrites the live index file of its replacement:

index after recreate + legitimate write C: (268 bytes, sha 376ae8b8a2a4)
index after write through the STALE handle: (424 bytes, sha 41d3bffdce74)
stale handle rewrote the live index file:  True
after restart, collection contains: B (stale write) = True, C (legitimate) = True

The stale write's row also lands in the replacement's records table, and its _PendingOperationRow carries the same (namespace, name) — and those rows are replayed on startup "applied or not", which is how the record survives a restart. _save_collection_index was likewise unscoped, so a stale handle could delete the replacement's applied pending operations and flip its index_saved, which marks the on-disk index as part of the durable contract.

Description

An incarnation is minted per creation, stored on _CollectionRow, and made part of everything that was previously keyed by (namespace, name):

• _collection_prefix, and so the records table and (for sqlite-vec) the vec0 virtual table
• the on-disk index path and the _search_engines cache key
• _PendingOperationRow.incarnation, the save-threshold count, the "mark applied" updates, and the startup replay, which now skips operations whose incarnation is not the collection's current one
• _save_collection_index, so its pending-operation delete and index_saved update touch only the incarnation that saved

A stale handle therefore addresses dropped tables and fails loudly rather than writing into the replacement.

This deliberately does not take a CollectionRegistry dependency. The registry exists to give a backend a transactional metadata authority it does not have; these stores already are one — _CollectionRow lives in the same engine as the per-collection tables and create_collection does the DDL and the metadata write in one transaction. Delegating would split one authority into two and reintroduce a crash window that the registry-backed stores accept only because they have no alternative.

Breaking changes

Existing databases are not migrated. Collections created by an earlier version keep resource names this version does not address, so their data is not reachable after upgrade. That is deliberate: migration is deferred along with the wider question of where the server runs DDL, and older data stays with older server versions for now.

Fixes/Closes

Fixes #1536.

Type of change

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

• Unit Test

Seven new tests across the two stores: a stale handle cannot write to a recreated collection; a stale handle does not clobber the recreated collection's index file, verified by bytes before and after and by reopening the database without a clean shutdown (a clean shutdown rewrites the index from the live engine and hides the damage); a database predating the column is migrated on startup and stays usable; and the unsuffixed name form is pinned as a compatibility contract.

Test Results: full server suite 1876 passed, 3 skipped; vector store suite 300 passed; ruff check, ruff format --check clean; ty check packages unchanged at 20 baseline diagnostics, none in the touched files.

Checklist

• I have signed the commit(s) within this pull request
• My code follows the style guidelines of this project (See STYLE_GUIDE.md)
• I have performed a self-review of my own code
• I have commented my code
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

Further comments

Independent of the collection-registry stack (#1526, #1527, #1530, #1531, #1533) and based on main so it can merge on its own. #1531 states the invariant this restores as a VectorStoreCollection contract and currently names these two stores as known non-conformance; whichever lands second should drop that note.

Terminology follows #1545, which introduces the same concept in the segment store: "incarnation" rather than "generation", since the value is a random uuid and generation/epoch invite the assumption that incarnations are ordered and comparable. The branch name still says generation; renaming it would break this PR's head ref, so it stays.

Two adjacent items are deliberately left out. SQLiteVecVectorStore.create_collection still reads, checks, then inserts although its primary key could arbitrate directly, which is a separate defect from this one. And a shared conformance test running this sequence against every VectorStore implementation would be worth having, but it needs the container-backed suites and belongs in its own change.