Repository navigation
[vector store alt 7/9] Remove custom sharding from the Qdrant store, and disable strict mode on its collections (speedkick) - #1642
Closed
edwinyyyu wants to merge 8 commits into
Conversation
Reverts the second commit of MemMachine#1606 (a8322a7), "Remove per-project filterable properties", and keeps its first, the OpenAPI regeneration under the locked FastAPI 0.141.1: docs/openapi.json is regenerated here with the same tool, so it differs from the pre-MemMachine#1606 document only by the ValidationError fields that regeneration added. A project declares `properties_schema`, a set of caller property keys with types, on its long-term memory configuration; the event backend merges it into the vector store collection's indexed schema and rejects filters on any other `m.<key>`. Under this design the schema is fixed for the life of the project, a schema change is a delete and a create, and the vector store maps every distinct schema to one physical collection that all projects declaring it share. What that costs, per backend, is written where a tenant will see it: a warning in the configuration docs and the field descriptions on the server configuration, the project API, the memory-configuration API and the Python SDK. Alternative to the one-collection chain (MemMachine#1622..MemMachine#1616): the two exclude each other. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…quest The event backend created a session's vector store collection and segment store partition on the first request that opened the session, so a search or a write for an unknown session created storage as a side effect, and the service locator was the only place that knew both stores' create paths. The owner is the session. Every path that creates a session row runs through EpisodicMemoryManager._create_session, which inserts the row and, when the row is new, creates the session's partitions in its segment store and its vector store (create_episodic_memory_storage); an equivalent re-create accepts the row and leaves the storage as it is. The request path binds handles with the stores' lookups and raises SessionPartitionMissingError when a partition is absent: a session without its storage is broken, not new. Deleting a session with no open instance deletes its partitions by key, so a session whose storage was never fully created can still be deleted. MemMachine.create_session goes through the manager for the same reason. The session's vector store collection is created with the system keys and the project's own `properties_schema`, resolved and merged as the request path did before. The semantic manager owns its one collection and creates it, once, at the storage's first use. With that, nothing calls the stores' open-or-create. The API is unchanged: the manager's open-or-create still creates a session a memory request names, now through the same path. Ported from MemMachine#1622 (4266b09) onto the chain that keeps per-project filterable properties. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The simple chatbot example, the TypeScript REST demo and the Dify plugin's add-memory tool wrote to a project without creating it, relying on the write to create it. Each now creates its project before its first memory request and accepts 409 as the project already existing. No behavior changes for them; they stop depending on a write creating a project. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
Adding memories to, or searching, a project that did not exist created it, with the server's default configuration, without the caller's knowledge. Now only the create-project request creates a project: a write or a search opens the session and answers 404 for an unknown project, as the search endpoint already promised; the manager's open-or-create goes. Two callers depended on the implicit creation. `org_id` and `project_id` default to `universal`, so the API promises the project `universal/universal`; the server creates it, once, at startup, and leaves one that already exists as it is. The MCP add tool names its own project and has no create-project counterpart, so it creates the project it writes to, once, and says so. The API doc strings and the OpenAPI document say which requests create a project. A breaking API change on `speedkick`. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
Nothing calls them since a session's storage is created with the session: `open_or_create_collection` and `close_collection` leave the vector store interface and its four backends, `open_or_create_partition` and `close_partition` leave the segment store interface and its implementation, and the two config-mismatch errors that only open-or-create raised go with them. A store creates on `create_*`, strictly, and looks up on `open_*`, answering None; create-if-absent is the owner's, where the key's provenance is known. Source changes are deletions only. The tests that exercised open-or-create as a fixture use a test-side create-if-absent instead, and the tests of its own semantics go. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
A lookup that answers None for an absent collection or partition is a Python get, not an open. Identifiers only, produced by the script below, plus the two docstring lines that said "opened"; every contract is as it was. ```sh set -e cd "$(git rev-parse --show-toplevel)" git ls-files -z 'packages/server/*.py' | xargs -0 perl -0pi -e ' s/\bopen_collection\b/get_collection/g; s/\bopen_partition\b/get_partition/g; ' uv run ruff check --fix --quiet packages/server uv run ruff format --quiet packages/server ``` The `open -> get` half of MemMachine#1626 (e9783d9); the `collection -> partition` half is not carried, because its premise is a store that is one collection. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`QdrantConf.is_distributed` put every logical collection in its own shard of the physical collection so that deleting one was a shard drop: an O(1) deletion in a cluster, at the price of a shard, with its own segments and graphs, per logical collection. The lifecycle that follows this change makes a deletion a registry write, visible at once, and reclaims the points afterward by a filter on the incarnation; the shard has nothing left to buy. The option goes from the configuration, the parameters, the database manager and the store, along with the shard-key selectors on every operation and the tests that ran a one-node cluster for it. A breaking configuration change on `speedkick`; the option was never documented. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A caller may filter on any property, declared in the collection's schema or not; an undeclared key is evaluated against the payload of the tenant's own candidate points, without an index. Qdrant Cloud enables strict mode on new collections by default, and strict mode rejects a filter on an unindexed payload key, so on such a deployment every undeclared filter failed. The store now creates its collections, the physical ones that hold points and the per-namespace registries, with `strict_mode_config` disabled. Collections created by an earlier version keep the setting they have. The chain that indexes every filterable key keeps strict mode on; this one, which lets a project declare what is indexed, turns it off. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This was referenced Sep 15, 2026
Contributor
Author
|
Reopen if stack chosen. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Alternative to the
[vector store N/13]chain (#1597, #1622..#1616): the two exclude each other. This chain keeps per-projectproperties_schema(user-defined metadata indexes) and conforms Qdrant to the incarnation lifecycle; SQLite and Milvus follow later. Every PR is a draft.Purpose of the change
Two commits.
QdrantConf.is_distributedput every logical collection in its own shard so that deleting one was a shard drop; with the next slice's lifecycle a deletion is a registry write and reclamation a filter on the incarnation, so the shard, with its own segments and graphs per logical collection, has nothing left to buy. The option goes from the configuration, the parameters, the database manager, the store and the tests that ran a one-node cluster for it.The store then creates every collection it makes, physical and registry (and, in the next slice, the purge queue), with
strict_mode_configdisabled: a caller may filter on any property, declared or not, and Qdrant Cloud's default strict mode rejects a filter on an unindexed payload key. The chain that indexes every filterable key keeps strict mode on; this one turns it off. Collections created by an earlier version keep their setting. Both are breaking configuration changes onspeedkick;is_distributedwas never documented.Stack
Slice 7 of 9, every PR targeting
speedkick; merge bottom-up.This PR's own change is its commits
c8baf723aand9ae55c3c5; the rest of its diff is the slices under it, and drops out as they merge. Stacked on slice 6; slice 8 is stacked on it.Verification
ruff check,ruff format --check,ty check,pytest packages/server/server_tests(integration deselected): server 1919 passed, 3 skipped; Qdrant integration lane (REST + gRPC) 122 passed; run at this slice's own commit on 2026-09-15.🤖 Generated with Claude Code