Skip to content

[vector store scale-out 1/5] Remove custom sharding from the Qdrant store (port of #1654) - #1671

Merged
edwinyyyu merged 1 commit into
MemMachine:mainfrom
edwinyyyu:port/qdrant-remove-custom-sharding-main
Sep 25, 2026
Merged

[vector store scale-out 1/5] Remove custom sharding from the Qdrant store (port of #1654)#1671
edwinyyyu merged 1 commit into
MemMachine:mainfrom
edwinyyyu:port/qdrant-remove-custom-sharding-main

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Port

Copy of #1654, merged into speedkick as 779362c61, on main: the same commit cherry-picked, with two differences for main: get, which main still has, also loses its shard-key selector, and the removed sharding tests are removed in main's form.

Purpose of the change

The Qdrant store could shard its native collection by logical collection (QdrantConf.is_distributed, CUSTOM sharding, one shard key per logical collection, a shard-key selector on every operation) so that a logical collection could be deleted by dropping its shard. The payload partition key was written and filtered in both modes, with the shard key on top, so no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured in #1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an explicit create_shard_key call of about 450 ms, 45 s for the hundred against 0.3 s with payload partitioning alone, including 10,000 points; 505 segments against 5, and still 5 after deleting and reusing tenants; and cluster mode is required, with a bootstrap --uri. At ten thousand tenants that is over an hour of shard-key creation and some 50,000 segments before a point is written. Qdrant's own guidance says the same: a physical structure per tenant is for the few oversized ones, the long tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing (a write to a dropped shard fails). Deletion is about to become a registry write that is O(1) and atomic as seen by every reader, a stale handle is fenced by that registry, and the points are reclaimed afterward by a filter-delete off the request path, so a shard per logical collection would only add its cost; it goes now, on the current shape, so the later changes do not carry it. Promoting a single oversized tenant to its own custom-sharded collection later, Qdrant's tiered arrangement, stays open: it is a separate collection, not a flag on this one.

is_distributed was never documented. A configuration that still names it is ignored, as any unknown key under a database config is (QdrantConf does not forbid extras); nothing reads it.

Stack

20 PRs on main: 2 independent ones, and three consecutive stages numbered on their own: the vector store scale-out, the SQLite store fixes and the vector store contract changes. A stacked PR's diff on GitHub is cumulative until the PRs under it merge.

Independent PRs, each directly on main; review and merge in any order:

# PR change
— #1541 fix(event-backend): make expand_context return timeline-neighbor episodes
— #1624 Make no memory request create a project

The vector store stages, consecutive: each stacked on the one below. [vector store scale-out 1/5] is directly on main and 2/5 on it alone; they do not depend on the independent PRs. From 3/5 up, the chain's history also carries the independent PRs' commits beneath it, since the later stages were written on top of them ([vector store contract 2/6] on #1624's session policy, for one); so a merge of an independent PR rebases the chain without conflict, and until they merge those PRs' changes show in a stacked PR's diff.

# PR change
Stage 1, the vector store scale-out: what horizontal scalability without sharding requires.
[vector store scale-out 1/5] #1671 (this PR) Remove custom sharding from the Qdrant store (port of #1654)
[vector store scale-out 2/5] #1631 Mint an incarnation per collection life in a SQL-arbitrated registry, so any process may serve any Qdrant or Milvus collection
[vector store scale-out 3/5] #1670 Remove per-project filterable properties (port of #1606)
[vector store scale-out 4/5] #1702 Keep user properties out of the vector store
[vector store scale-out 5/5] #1627 Make a vector store one collection, with string-keyed partitions
Stage 2, the SQLite store fixes, on [vector store scale-out 5/5]: re-derived on the one-collection store.
[sqlite store fixes 1/7] #1460 Publish vector index files atomically (but not durably) (as merged into speedkick, #1588)
[sqlite store fixes 2/7] #1469 Never reuse a row id in SQLiteVectorStore (as merged into speedkick, #1589)
[sqlite store fixes 3/7] #1672 Own the search engine's concurrency in the store, not in each engine (port of #1612)
[sqlite store fixes 4/7] #1673 Serialize a partition's writes so the engine sees them in order (port of #1607)
[sqlite store fixes 5/7] #1674 Refuse a pending row replay cannot honor, instead of dropping it (port of #1608)
[sqlite store fixes 6/7] #1675 Take SQLite's write lock at BEGIN, not at the first write (port of #1609)
[sqlite store fixes 7/7] #1676 Give every write a fresh row id, so a key names one version (port of #1610)
Stage 3, the vector store contract changes, on [sqlite store fixes 7/7].
[vector store contract 1/6] #1663 Answer with cosine scores and uuids, not vectors and stale properties, and require a vector on the record (port of #1598 and #1603)
[vector store contract 2/6] #1622 Create a session's storage with the session, never on a request
[vector store contract 3/6] #1625 Remove open-or-create and close from both stores
[vector store contract 4/6] #1618 Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization
[vector store contract 5/6] #1628 Make a vector store filter only on the properties it declares
[vector store contract 6/6] #1616 Close the filter union, and make negation the complement on every backend

This PR is its one commit, 56beebe79, directly on main; #1631 is stacked on it.

Verification

At its one commit, on 2026-09-25: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); the full server suite without integration tests passes (pytest packages/server/server_tests -m "not integration"), 1982 tests at its head 56beebe79.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

This was referenced Sep 16, 2026
Merged
Closed
Merged
Closed
Merged
Open
Open
Draft
Closed
Open
Closed
Closed
Closed
Closed
Closed
Open
Closed
Draft
Closed
Draft
Draft
Closed
Draft
Closed
Closed
Draft
@edwinyyyu
edwinyyyu force-pushed the port/qdrant-remove-custom-sharding-main branch from 2ed90a0 to 0deebeb Compare September 17, 2026 17:39
@edwinyyyu edwinyyyu changed the title [speedkick port 17/17] Remove custom sharding from the Qdrant store (port of #1654) [vector store 5/16] Remove custom sharding from the Qdrant store (port of #1654) Sep 17, 2026
This was referenced Sep 17, 2026
Draft
Draft
@edwinyyyu edwinyyyu changed the title [vector store 5/16] Remove custom sharding from the Qdrant store (port of #1654) Remove custom sharding from the Qdrant store (port of #1654) Sep 17, 2026
edwinyyyu
edwinyyyu commented Sep 17, 2026
View reviewed changes

@edwinyyyu edwinyyyu left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. "is_distributed was never documented; a configuration naming it is rejected": the second half does not hold. SupportedDB.build_config does conf_cls(**conf), no configuration model sets extra="forbid", and pydantic's default is extra="ignore", so SupportedDB.QDRANT.build_config({"host": "h", "registry_database": "r", "is_distributed": True}) returns a QdrantConf with no error: the key is silently ignored. Same sentence on the origin, #1654.
  2. The commit trailer "cherry picked from 408b0a9" names a local chain-main3 commit; #1654's merge commit is 779362c.

@edwinyyyu edwinyyyu mentioned this pull request Sep 17, 2026
Merged
@edwinyyyu
edwinyyyu force-pushed the port/qdrant-remove-custom-sharding-main branch 2 times, most recently from 9bd0467 to 2036379 Compare September 18, 2026 19:39
This was referenced Sep 18, 2026
Draft
Draft
@edwinyyyu
edwinyyyu force-pushed the port/qdrant-remove-custom-sharding-main branch from 2036379 to f2436a7 Compare September 21, 2026 18:06
@edwinyyyu edwinyyyu mentioned this pull request Sep 21, 2026
Open
@edwinyyyu
edwinyyyu marked this pull request as ready for review September 21, 2026 22:16
@edwinyyyu edwinyyyu changed the title Remove custom sharding from the Qdrant store (port of #1654) [vector store scale-out 1/5] Remove custom sharding from the Qdrant store (port of #1654) Sep 21, 2026
marvinyu-memverge
marvinyu-memverge approved these changes Sep 22, 2026
View reviewed changes

@marvinyu-memverge marvinyu-memverge left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at f2436a7. One non-blocking ask, one observation.

  1. The commit message still carries the origin's sentence "is_distributed was never documented; a configuration naming it is rejected", which your own self-review on 09-17 corrected: no conf model sets extra="forbid" (0 hits for extra= or model_config under common/configuration/ at head), so build_config's conf_cls(**conf) silently drops the key. The PR body already says "ignored"; the commit is what git log keeps. Worth amending to the body's wording on the next push, and the title too, since the commit still says "[vector store 2/13]" where the PR is "[vector store scale-out 1/5]".

Observation, not an ask: a collection created under is_distributed=True keeps CUSTOM sharding on the Qdrant side, and Qdrant requires a shard-key selector on an upsert to a custom-sharded collection, so a deployment that had set the flag would fail every write after this lands until it recreates its collections. The flag was undocumented and needed cluster mode with a bootstrap --uri, so I do not expect a producer; noting it in case an upgrade note exists for the speedkick deployment.

Verified: diff-of-diffs against 779362c (git diff 779362c^ 779362c vs origin/main..refs/pr/1671, changed lines sorted) differs in exactly the two stated adaptations: get's shard_key_selector, and the removed sharding tests in main's form (coll.get / record.uuid / the COSINE lines where the origin removed _present_uuids). git grep at head for is_distributed|shard_key|_ensure_shard_key|ShardingMethod|distributed_qdrant: 0 hits repo-wide, docs and sample configs included (main: 37 hits, all inside this PR's seven files). delete_collection now takes the pre-existing FilterSelector(_partition_filter(name)) path unconditionally. CI green at head, no threads.

@edwinyyyu @claude
[vector store 2/13] Remove custom sharding from the Qdrant store (spe…
56beebe 
…edkick) (MemMachine#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in MemMachine#1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 779362c)
@edwinyyyu
edwinyyyu force-pushed the port/qdrant-remove-custom-sharding-main branch from f2436a7 to 56beebe Compare September 25, 2026 18:03
@edwinyyyu
edwinyyyu merged commit d556f4b into MemMachine:main Sep 25, 2026
44 checks passed
@marvinyu-memverge marvinyu-memverge mentioned this pull request Sep 25, 2026
Merged
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Oct 1, 2026
@edwinyyyu @claude
Keep the design documents to the design
ba644cc 
From an audit against the PR's writing rules, across the seven vector
store design documents:
- They narrated changes and PRs (what "is gone", what the configuration
  "gains" and "loses", what MemMachine#1627, MemMachine#1663, MemMachine#1625, MemMachine#1661 and MemMachine#1671 do, and
  "the previous design"), and claimed MemMachine#1468 fixed by two PRs that are
  open. Each now states the design, keeping links to open issues.
- They had drifted from the code: a delete checks liveness once, after
  its call; Qdrant has no load step and Milvus loads on every
  preparation; the handle's writes run with qdrant-client's default
  wait rather than passing it; the Qdrant store states no replicated
  read delay; and the purge contract promises bounded work per call,
  while oldest-first rounds are the registry-backed stores'.
- Repetition across documents goes: the clients section, the replicated
  Qdrant measurements and the Qdrant id alternatives now live in one
  document each, linked from the others; consistency cites the Strong
  wait's measurement instead of a figure without its conditions; the
  async-client measurement names its Milvus version.
- The UUID text form, a string format rather than a design decision,
  is gone from the isolation document.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This was referenced Oct 1, 2026
Merged
Merged
Merged
Merged
Draft
Merged
Draft
Merged
Open
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@malatewang malatewang malatewang approved these changes
@marvinyu-memverge marvinyu-memverge marvinyu-memverge approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@edwinyyyu @malatewang @marvinyu-memverge