[Feat]: Cross-process collection registry for vector store metadata #1524

Status 2026-10-02. Addressed by #1734 (SQL collection registry: create by primary key, an incarnation per collection life, purge claimed with SKIP LOCKED) with the Qdrant move in #1735 and the Milvus move in #1736; closes on merge. #1525 was a Qdrant-specific statement of the same request and is closed as its duplicate.

Is your feature request related to a problem?

Horizontal scalability: multiple MemMachine server processes should be able to work with the same collections and partitions on one backend (e.g. a Qdrant cluster). Strict sharding — each collection managed by exactly one process — is what the VectorStore contract requires today ("must be managed by at most one process at a time", vector_store.py), and it works; the restriction itself is the limitation. What prevents lifting it is collection metadata management, not the data path: each vector store keeps its own ad-hoc catalog of logical collections (Qdrant and Milvus in dummy-vector __registry collections inside the backend itself, the SQLite stores in bespoke tables), and for backends with no conditional writes, transactions, or unique constraints there is no way to make create / open-or-create / delete atomic — they are read-check-write sequences guarded only by per-process asyncio locks.

Describe the solution you'd like

A generic, cross-process-safe config registry primitive that collection metadata can be moved onto:

• ConfigRegistryManager — provides named registries over one backing store; the isolation guarantee (different names -> disjoint entries; same name -> same entries, across processes) is part of the contract; open_registry(name, config_adapter) idempotently prepares storage and returns a scoped handle.
• ConfigRegistry[ConfigT] — the handle: create (atomic, fails on existing key), get, get_or_create (atomic; returns the stored config and whether this call created it; never compares configs — equality policy stays with the consumer), idempotent delete. No name parameter on any operation, so a consumer holding a handle structurally cannot reach other registries.
• A SQLAlchemy implementation (PostgreSQL + SQLite) using one table per registry with the primary key as the compare-and-set arbiter.

First consumer: QdrantVectorStore's collection registry (tracked separately). Later candidates: MilvusVectorStore's __registry, the SQLite vector stores' _CollectionRow tables, the segment store's PartitionRow, and the session manager's SessionConfig (currently a race-prone read-then-insert).

Describe alternatives you've considered

• Keeping the catalog inside each vector database: Qdrant/Milvus offer no CAS primitive, so the races are structurally unfixable there.
• Bespoke per-backend SQL tables: duplicates the subtle insert-conflict handling once per backend instead of once.
• A generic key->config registry (an earlier draft of this proposal): speculative surface with one consumer kind; the collection-specific API says the same things in domain terms and can be generalized later if a non-collection consumer materializes.
• Distributed locks/leases: heavier operationally, and crash-recovery semantics are worse than a unique-constraint insert that is atomic by construction.

Additional context

An implementation PR follows immediately; further PRs switch QdrantVectorStore and MilvusVectorStore onto it.

🤖 Written by Claude Code (Opus 5) on behalf of @edwinyyyu.