[Feat]: Redesign QdrantVectorStore collection registry for multi-process collection management #1525

Is your feature request related to a problem?

Horizontal scalability: multiple MemMachine server processes should be able to work with the same logical collections on one Qdrant cluster. Strict sharding — each collection managed by exactly one process, names partitioned across processes — is what the VectorStore contract requires today ("must be managed by at most one process at a time"), and it works. The restriction itself is the limitation, and it cannot be lifted with the current registry design.

Collection metadata lives in per-namespace <ns>__registry Qdrant collections (dummy-vector points keyed uuid5(name)). Qdrant has no conditional writes, transactions, or unique constraints, so create_collection / open_or_create_collection / delete_collection are non-atomic read-check-write sequences whose only guard is a per-process asyncio lock. The moment two processes manage the same name, the contract's guarantees dissolve:

1. VectorStoreCollectionAlreadyExistsError stops firing — two concurrent create_collection calls both pass the absence check and last-writer-wins the registry point.
2. Two open_or_create_collection calls with different configs each create a different native collection (native names are sha256(config) while registry points are keyed by name), one silently wins the registry point, and records written through the losing handle become permanently unreachable — VectorStoreCollectionConfigMismatchError never fires.

So this is not a bug under the current contract; it is the reason the contract has to forbid sharing, and lifting it requires redesigning where the registry lives.

Describe the solution you'd like

Move QdrantVectorStore's collection registry out of Qdrant onto the cross-process config registry primitive (#1524), whose unique-constraint insert is a real compare-and-set:

• The registry insert becomes the atomic commit point of collection creation (native-collection creation first, registration last, so a crash leaves only an empty config-shared native collection that the next same-config creation adopts).
• AlreadyExists / ConfigMismatch guarantees then hold across processes sharing the same registry database; the per-process lock remains only as in-process serialization.
• Native collection naming is unchanged.

Describe alternatives you've considered

• Keeping the registry inside Qdrant: structurally unfixable — there is no CAS primitive to build atomic create on.
• Distributed locks/leases around lifecycle operations: operationally heavier, and crash-recovery semantics are worse than an insert that is atomic by construction.

Additional context

Scope: collection lifecycle/metadata management. Registry entries store resolved identity (native collection name, plus a generation-scoped partition key), which also makes delete_collection safe against handles held in other processes: writes through a held handle land under the dead generation — invisible, never resurrected by a re-creation. Read-after-write visibility tuning on the data path remains a separate concern.

Companion redesign pieces: strict create/open lifecycle (open-or-create removal) and a declared per-instance concurrency scope (PROCESS/MACHINE/CLUSTER) replacing the ABC's blanket one-process-per-collection sentence, so the capability is expressible through the contract.

🤖 Written by Claude Code (Opus 5) on behalf of @edwinyyyu.