Skip to content

Accept attempts exhausted in the lifecycle churn contract, and say which Qdrant operations filter on the incarnation - #1775

Merged
edwinyyyu merged 2 commits into
MemMachine:feat/horizontal-scalingfrom
edwinyyyu:fix/vector-store-qdrant-followups-speedkick
Oct 7, 2026
Merged

edwinyyyu merged 2 commits into
MemMachine:feat/horizontal-scalingfrom
edwinyyyu:fix/vector-store-qdrant-followups-speedkick

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the change

Summary

Two corrections to #1735. Independent of #1736; either can merge first.

  • Churn contract: test_lifecycle_churn_raises_only_domain_errors now accepts VectorStoreAttemptsExhaustedError, which create_collection and open_or_create_collection document as the outcome of creation races that make no progress. The Qdrant store's own churn test already accepts it; the contract's test could fail on a documented outcome.
  • Qdrant design: the layout section said every search, scroll, and delete of a handle filters on the incarnation. A handle's deletes name its points' derived ids, with no filter, and the scrolls and filtered deletes are the purge's. The document now says so, pointing to the point id section.

Commits

  1. Accept an attempts-exhausted error in the lifecycle churn contract.
  2. Say which Qdrant operations filter on the incarnation.

Stack

19 open PRs: one independent PR, and the vector store tree of short parallel branches. #1736's and #1775's GitHub base is feat/horizontal-scaling and the others' is main, since the branches are in a fork and a pull request can target only this repository's branches; the on column gives the order the PRs build on each other instead. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

# PR change on
— #1624 Make no memory request create a project main

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1670 and #1702 sit beneath #1627, whose code depends on them. Until the PRs under it merge, their changes show in a stacked PR's diff.

# PR change on
[vector store scale-out 1/6] #1671 (merged) Remove custom sharding from the Qdrant store (port of #1654) main
[vector store scale-out 2/6] #1733 (merged into feat/horizontal-scaling) Answer vector store queries with record UUIDs and scores, and refuse invalid inputs feat/horizontal-scaling
[vector store scale-out 3/6] #1734 (merged into feat/horizontal-scaling) Arbitrate vector store collections in a SQL registry, with an incarnation per collection life and a purge feat/horizontal-scaling
[vector store scale-out 4/6] #1735 (merged into feat/horizontal-scaling) Move the Qdrant store onto the collection registry feat/horizontal-scaling
[vector store scale-out 5/6] #1736 Move the Milvus store onto the collection registry, against a Milvus server feat/horizontal-scaling
— #1775 (this PR) Accept attempts exhausted in the lifecycle churn contract, and say which Qdrant operations filter on the incarnation feat/horizontal-scaling
— #1631 (closed) Superseded by #1733–#1736, which hold its changes split in four, with review changes since —
[user properties 1/2] #1670 Remove per-project filterable properties (port of #1606) #1736
[user properties 2/2] #1702 Keep user properties out of the vector store #1670
[vector store scale-out 6/6] #1627 Make a vector store one collection, with string-keyed partitions #1702
[session storage 1/2] #1622 Create a session's storage with the session, never on a request #1627
[session storage 2/2] #1625 Remove open-or-create from both stores, and close from the segment store #1622
[search results] #1663 Score every vector search by cosine similarity, and name scores for it (port of #1598's cosine half) #1627
[sqlite store fixes 1/7] #1460 Publish vector index files atomically (but not durably) #1663
[sqlite store fixes 2/7] #1469 Never reuse a row id in SQLiteVectorStore #1460
[sqlite store fixes 3/7] #1672 Own the search engine's concurrency in the store, not in each engine (port of #1612) #1469
[sqlite store fixes 4/7] #1673 Serialize a partition's writes so the engine sees them in order (port of #1607) #1672
[sqlite store fixes 5/7] #1674 Refuse a pending row replay cannot honor, instead of dropping it (port of #1608) #1673
[sqlite store fixes 6/7] #1675 Take SQLite's write lock at BEGIN, not at the first write (port of #1609) #1674
[sqlite store fixes 7/7] #1676 Give every write a fresh row id, so a key names one version (port of #1610) #1675
[qdrant options] #1618 Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization #1663
[declared schema 1/2] #1628 Make a vector store filter only on the properties it declares #1663
[declared schema 2/2] #1616 Close the filter union, and make negation the complement on every backend #1628

This PR is its 2 commits, 1218f18dc, ba9f952db, directly on feat/horizontal-scaling.

Verification

At the head: ruff check, ruff format --check, and ty check are clean; the full server suite without integration tests passes (2076 passed, 8 skipped); in test containers, the integration tests pass (1453 passed, 266 skipped), the churn tests among them against Qdrant 1.19.1 over REST and gRPC. Commit 2 changes documentation only.

🤖 Generated with Claude Code

edwinyyyu and others added 2 commits October 6, 2026 17:25
create_collection and open_or_create_collection document
VectorStoreAttemptsExhaustedError as the outcome of creation races that
make no progress, and the Qdrant store's own churn test already accepts it.
The contract's churn test refused it, so it could fail on a documented
outcome.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The design said every search, scroll, and delete of a handle filters on the
incarnation. A handle's deletes name its points' derived ids, with no
filter; the scrolls and filtered deletes are the purge's.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@edwinyyyu edwinyyyu added the horizontal scaling Wrong or unsafe when more than one server process serves the same backends (replicas or workers) label Oct 7, 2026
This was referenced Oct 7, 2026
@edwinyyyu
edwinyyyu merged commit d45d8dd into MemMachine:feat/horizontal-scaling Oct 7, 2026
39 checks passed
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Oct 7, 2026
…ich Qdrant operations filter on the incarnation (MemMachine#1775)

* Accept an attempts-exhausted error in the lifecycle churn contract

create_collection and open_or_create_collection document
VectorStoreAttemptsExhaustedError as the outcome of creation races that
make no progress, and the Qdrant store's own churn test already accepts it.
The contract's churn test refused it, so it could fail on a documented
outcome.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Say which Qdrant operations filter on the incarnation

The design said every search, scroll, and delete of a handle filters on the
incarnation. A handle's deletes name its points' derived ids, with no
filter; the scrolls and filtered deletes are the purge's.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
@edwinyyyu edwinyyyu mentioned this pull request Oct 7, 2026
26 tasks
@edwinyyyu edwinyyyu removed the horizontal scaling Wrong or unsafe when more than one server process serves the same backends (replicas or workers) label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant