[vector store 11/14] Keep the Qdrant and Milvus partition registries in SQL, so any process may create, delete and purge (speedkick) - #1656

The simple chatbot example, the TypeScript REST demo and the Dify plugin's
add-memory tool wrote to a project without creating it, relying on the
write to create it. Each now creates its project before its first memory
request and accepts 409 as the project already existing. No behavior
changes for them; they stop depending on a write creating a project.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Adding memories to, or searching, a project that did not exist created it,
with the server's default configuration, without the caller's knowledge.
Now only the create-project request creates a project: a write or a
search opens the session and answers 404 for an unknown project, as the
search endpoint already promised; the manager's open-or-create goes.

Two callers depended on the implicit creation. `org_id` and `project_id`
default to `universal`, so the API promises the project
`universal/universal`; the server creates it, once, at startup, and leaves
one that already exists as it is. The MCP add tool names its own project
and has no create-project counterpart, so it creates the project it writes
to, once, and says so. The API doc strings and the OpenAPI document say
which requests create a project.

A breaking API change on `speedkick`.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Nothing calls them since a session's storage is created with the session:
`open_or_create_collection` and `close_collection` leave the vector store
interface and its four backends, `open_or_create_partition` and
`close_partition` leave the segment store interface and its
implementation, and the two config-mismatch errors that only open-or-create
raised go with them. A store creates on `create_*`, strictly, and looks up
on `open_*`, answering None; create-if-absent is the owner's, where the
key's provenance is known.

Source changes are deletions only. The tests that exercised
open-or-create as a fixture use a test-side create-if-absent instead, and
the tests of its own semantics go.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…res' lookup to get_partition

A vector store's logical collection becomes a partition, the segment store's word for the same thing, and both stores' lookup is get_partition, answering None like a Python get. Identifiers only, produced by the script below; the (namespace, name) identity, the per-partition config and every docstring are as they were, and the next change gives them their meaning. The native clients' create_collection and delete_collection keep their names.

```sh
set -e
cd "$(git rev-parse --show-toplevel)"
git mv packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_collection.py \
       packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_partition.py
git ls-files -z 'packages/server/*.py' | xargs -0 perl -0pi -e '
  s/VectorStoreCollection(?!Config)/VectorStorePartition/g;
  s/in_memory_vector_store_collection/in_memory_vector_store_partition/g;
  s/vector_store_collection(?!_schema|_namespace)/vector_store_partition/g;
  s/open_collection/get_partition/g;
  s/def create_collection\(/def create_partition(/g;
  s/def delete_collection\(/def delete_partition(/g;
  s/\.create_collection\((\s*namespace=)/.create_partition($1/g;
  s/\.delete_collection\((\s*namespace=)/.delete_partition($1/g;
  s/\.create_collection(?=\s*=\s*AsyncMock|\.assert_)/.create_partition/g;
  s/\.delete_collection(?=\s*=\s*AsyncMock|\.assert_)/.delete_partition/g;
  s/"create_collection"/"create_partition"/g;
  s/"delete_collection"/"delete_partition"/g;
  s/only delete_collection is invoked/only delete_partition is invoked/g;
  s/test_delete_collection_/test_delete_partition_/g;
  s/open_partition/get_partition/g;
'
uv run ruff check --fix --quiet packages/server
uv run ruff format --quiet packages/server
```

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per
partition, a shard-key selector on every operation) so that a partition
could be deleted by dropping its shard. Partition deletion is about to
become a registry write that is O(1) and atomic as seen by every reader,
with the points reclaimed afterward by a filter-delete, so a shard per
partition would only add overhead; it goes now, on the current shape,
so the incarnation change does not carry it. `is_distributed` was never
documented; a configuration naming it is rejected.
`registry_replication_factor` stays.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The Qdrant and Milvus clients were built without a timeout, so a remote
write could hang a request indefinitely. `request_timeout` on QdrantConf
and MilvusConf, in seconds, is passed to the client; it is required, with
no default, so a deployment states how long it is willing to wait, and the
configuration wizard supplies 30 seconds as the starting point. The sample
configurations and the configuration docs show the option.

A breaking configuration change on `speedkick`.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…uilt by the composition root

A vector store was a factory of logical collections, each identified by a
(namespace, name) pair and created with its own dimensions and schema; a
backend that limits native collections shared one among logical
collections of equal configuration, under a name derived from a hash of
that configuration, and a registry per namespace mapped names to it.

A store is now one collection: `VectorStore(collection, vector_dimensions,
indexed_properties)` names its one native collection (or its tables and
index files) at construction, every partition of it shares the
collection's dimensions and schema, and `provision()` creates the
collection's durable resources idempotently, before `startup`.
`create_partition(key)`, `get_partition(key)` and `delete_partition(key)`
take a string key; a partition is a payload value (Qdrant), a partition-key
value (Milvus) or a pair of tables (the SQLite stores) inside the
collection, and the registry beside it records what each partition was
created under, so a store built with other dimensions or another schema
raises VectorStorePartitionSchemaMismatchError instead of reading columns
and vectors that are not there. Collection names may be 64 bytes; the
hash-derived native names go, and with them `VectorStoreCollectionConfig`
and the per-partition config.

`DatabaseManager.get_vector_store(backend, collection=, vector_dimensions=,
indexed_properties=)` builds and caches one store per (backend,
collection), keyed by the service's system keys; asking for a collection
again with other dimensions or keys is a configuration error. The event
backend's collection is `long_term_memory__<embedder>` and the semantic
memory's `semantic_memory__<embedder>`, one cell of the purpose-by-embedder
matrix each; the two SQLite stores of one backend share its engine, and
MemMachine warms the event backend's store through the locator, since
building it needs the embedder's dimensions.

The data path is as it was: a partition stores every property of a record
and filters on any key, with the declared keys indexed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…purge

A partition is identified to callers by its key and inside every store by
an incarnation the store mints when the partition is created. Records,
points and index files are keyed by the incarnation, never by the key, so
a partition deleted and re-created under the same key starts empty and its
predecessor's storage is never adopted by, or reclaimed out from under,
the successor. Handles are bound to one incarnation: once it is deleted,
every operation of the handle raises VectorStorePartitionHandleStaleError.

delete_partition becomes a registry write that makes the partition
unreachable at once; the new purge_deleted_partitions reclaims the storage
afterward, oldest deletion first, a bounded amount per call, safe to repeat
and to run from several processes. Both SQLite stores keep every partition
of a collection in shared tables (records, vec0 with the incarnation as its
partition key, pending-operation log) with a purge queue beside the
registry, and fence writes with a self-checking UPDATE or a registry SELECT
under BEGIN IMMEDIATE; Qdrant and Milvus carry the incarnation in the
payload/partition-key field, keep purge entries in the registry collection
stamped with the deletion time, and fence each operation with a registry
lookup. create_partition mints under the segment store's rules: a
collision with a live or queued incarnation re-mints, up to
_MAX_MINT_ATTEMPTS, then VectorStoreAttemptsExhaustedError.

The SQLite stores' on-disk layout changes (shared tables per collection in
place of tables per partition); existing SQLite vector store files are not
migrated.

partition_lifecycle_contract.py holds the contract tests every backend
mixes in: stale handles, empty re-creation, idempotent deletion, purge
reclaiming what deletion deferred and leaving live partitions alone; run
against Qdrant in local, REST and gRPC modes.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The first time a vector store is handed out, the resource manager starts
the same purge loop it runs for segment stores, one per (backend,
collection); close() cancels both sets. Mechanical churn in the same
change: the loop's interval and pause constants lose their SEGMENT_STORE_
prefix and the loop takes a label for its failure log line.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…s may create, delete and purge

Qdrant and Milvus have no transactions, unique constraints or conditional
writes, so the registry each store kept inside its backend could not
arbitrate two server processes creating, deleting or reclaiming the same
partition: a create was read-then-write under a per-process lock, two
creators both succeeded with the last registration winning (Milvus, whose
insert does not enforce primary keys, kept both), and a delete was two
unrelated writes. The stores now keep their registry in the deployment's
relational database, `SqlPartitionRegistry`: one table pair per backend
kind (`vector_store_qdrant_pt`/`_gc`, `vector_store_milvus_pt`/`_gc`),
shared by every store on that kind of backend, a row per live partition
keyed by collection and partition key with the incarnation its points
carry, and a purge queue of dead incarnations. Creation is an insert the
primary key arbitrates (a racing creator on any process gets
AlreadyExists); deletion is one transaction, a DELETE that takes the row's
write lock and the queue insert; a purge claim is a row lock held while
the backend deletes the points, `FOR UPDATE SKIP LOCKED` on PostgreSQL so
concurrent purgers split a backlog, retired when the delete returns and
kept when it raises; the per-operation fence is a registry lookup by
incarnation. The registry collections, the deterministic registry point
ids, the deletion stamps and the per-process lock tables go, and with them
`registry_replication_factor`.

`QdrantConf` and `MilvusConf` gain a required `registry_database`, the
name of a relational database under `resources.databases`; DatabaseManager
hands its engine to the store, and `provision()` creates the two tables on
it, surviving a racing provisioner. The SQLite stores already keep their
registry beside their data in one file and are unchanged.

The `VectorStore` contract no longer restricts a partition to one process:
every operation is safe from any process sharing the backend, and a store
that cannot give that says so itself (the engine-backed SQLite store holds
a partition's index in the process that opened it; the sqlite-vec store is
shared by the processes of one node).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn