Feat: Back QdrantVectorStore collection metadata with SQL collection registry (collection registry stack 2/5) - #1527

Purpose of the change

Horizontal scalability: let multiple MemMachine server processes work with the same logical collections on one Qdrant cluster (#1525). Strict sharding of collection names across processes — what the VectorStore contract requires today — has always worked; this PR is the redesign that lifts the restriction for collection lifecycle management, by moving QdrantVectorStore's collection metadata out of Qdrant onto the SQL-backed config registry from #1526.

Stacked on #1526; only the last commit is new to this PR.

Qdrant offers no conditional writes, transactions, or unique constraints, so the current <ns>__registry bookkeeping (dummy-vector points keyed uuid5(name)) makes create_collection / open_or_create_collection / delete_collection non-atomic read-check-write sequences, guarded only by a per-process asyncio lock. The moment two processes manage the same name:

• VectorStoreCollectionAlreadyExistsError stops firing — two concurrent creates both pass the absence check and last-writer-wins the registry point.
• Worse, two open_or_create_collection calls with different configs each create a different native collection (native names are sha256(config) while registry points are keyed by name), one silently wins the registry, and records written through the loser's handle become permanently unreachable. VectorStoreCollectionConfigMismatchError never fires.

With the registry's unique-constraint CAS as the commit point, both guarantees hold across processes sharing the same registry database, and generationed partition keys make delete_collection safe against handles held in other processes. Scope: lifecycle/metadata management — data-path read-after-write visibility tuning remains a separate concern.

Description

qdrant_vector_store.py: all __registry machinery is deleted (_REGISTRY_* constants, uuid5 point ids, _ensure_namespace_registry_collection, _get_registry_entry, _parse_entry, _register_collection, _is_not_found_error, registry_replication_factor). QdrantVectorStoreParams gains a required registry: CollectionRegistry.

Registry entries store resolved identity, not just config (CollectionRegistryEntry from #1526): config, native_collection_name, and a generation-scoped partition_key (minted at creation; also the shard key in distributed mode).

• Native name pinned at creation: handles are built from the stored name rather than re-deriving sha256(config) on every open, so a later change to the config model's serialization (any added field re-hashes every config) cannot silently repoint existing collections at new empty native collections. The naming scheme for new collections is untouched.
• Generationed partition keys make deletion safe against held handles: delete_collection removes that generation's data and the entry; a handle held across the deletion keeps writing into the dead generation — invisible to every reader, storage-only garbage — and a re-creation mints a fresh generation, so stale records are never resurrected. No locks and no per-write round trips; the residual cost is bounded garbage under dead generations (a future GC can sweep partition keys absent from the registry).

Lifecycle ordering, chosen for crash windows:

• Create: native-first, register-last. The registry insert is the atomic commit point. A crash after native creation leaves only an empty native collection, which is shared by config and adopted by the next creation with the same config. (Claim-registry-first would be worse: open_collection builds handles straight from the entry, so a crash between claim and native create would hand out handles to a nonexistent native collection.) Invariant: registry entry implies native collection exists.
• Delete: data-first, unregister-last (unchanged ordering). A crash in between leaves a registered-but-empty collection (documented on delete_collection); retrying the delete completes it.
• Create-race losers leave an empty native collection (created before the CAS) and, in distributed mode, an unused shard key. Cleanup is deliberately not attempted — native collections are config-shared, so nothing can safely decide no other logical collection references one — and the residue is bounded by the number of distinct configs attempted.

_name_locks stays, with its comment updated: it now only serializes lifecycle operations within a process (preserving today's single-process open-vs-delete interleaving semantics and avoiding redundant idempotent round trips); cross-process correctness comes from the registry CAS.

Wiring (database_manager.py): async_get_qdrant_client builds and starts a SQLAlchemyCollectionRegistry named qdrant_<conf name> on the engine named by the new QdrantConf.registry_database, passing it into the store — QdrantVectorStore holds a registry dedicated to it and cannot reach any other. One registry per Qdrant conf entry, so two Qdrant instances sharing one registry database get separate tables. Misconfiguration (unknown database name, or a conf entry name that cannot form a registry table name) raises QdrantConfigurationError with the fix spelled out.

Configuration (database_conf.py, samples, docs): QdrantConf.registry_database (required) names a configured relational database (postgres or sqlite entry); registry_replication_factor is removed — it existed to give the Qdrant-hosted registry read-your-writes, which is moot now. Sample configs and the configuration docs gained the new field; the installation wizard points Qdrant at its existing SQLite entry.

Design doc: design/collection_registry_backends.md.

Breaking changes

• QdrantConf.registry_database is required: existing Qdrant users add one line of YAML (single-node setups can point at a sqlite relational entry). There is deliberately no per-host auto-SQLite fallback — divergent per-host registries would look consistent while reproducing exactly the races this PR removes. Stale registry_replication_factor YAML keys are silently ignored (extra="ignore").
• Existing <ns>__registry Qdrant collections are no longer read, and there is no data migration. This is mostly self-healing: native collection names are unchanged, so the first open_or_create_collection with an unchanged config re-registers it and all data is reachable again. Caveat: paths that re-create with a currently derived schema (e.g. the episodic event backend) will orphan old partitions if that schema drifted since original creation — the old registry would have preserved the original config. Stale __registry collections can be deleted manually.

Fixes/Closes

Fixes #1525

Type of change

• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?

• Unit Test

• Integration Test

• Existing QdrantVectorStore suite unchanged in behavior, now parametrized with a registry over the full client matrix: in-memory (unit) plus HTTP, gRPC, and distributed-cluster containers (integration).

• New TestRegistryIntegration: no __registry-suffixed collections exist in Qdrant after creation; two stores sharing one registry+client see each other's collections (create raises AlreadyExists cross-store, delete in one is observed by the other); concurrent open_or_create_collection with mismatched configs yields exactly one winner and one VectorStoreCollectionConfigMismatchError; ("a__b", "c") and ("a", "b__c") are distinct collections (key-ambiguity regression); re-creation mints a fresh generation over the same native collection; records upserted through a handle held across a deletion are unreachable from the re-created collection (resurrection regression).

• test_database_manager.py: registry built on the named engine and passed into the store; unknown registry_database and un-usable conf entry names raise QdrantConfigurationError and close the client.

• test_database_conf.py: field swap, required-field validation.

Test Results: full server suite 1901 passed locally; targeted integration run (qdrant containers + PostgreSQL registry) 452 passed; ruff check, ruff format --check, ty check packages clean (no new diagnostics).

Checklist

• I have signed the commit(s) within this pull request
• My code follows the style guidelines of this project (See STYLE_GUIDE.md)
• I have performed a self-review of my own code
• I have commented my code
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

Further comments

The same swap for MilvusVectorStore lands later in this stack.