[vector store scale-out 5/6] Move the Milvus store onto the collection registry, against a Milvus server by edwinyyyu · Pull Request #1736 · MemMachine/MemMachine

edwinyyyu · 2026-10-01T20:23:49Z

Purpose of the change

Summary

The Milvus store moves onto the collection registry, against a Milvus server, so any number of server processes sharing its registry may serve its collections. The store was rewritten rather than adapted, so the move is one commit; its message lists each change.

Catalog: the collection registry in the relational database MilvusConf.collection_registry names (required). The registry collections, whose insert let two creators of one name both succeed, and the process-local locks go.
Schema:
- The incarnation is the partition key, with partitionkey.isolation, and part of the primary key, "{incarnation}:{record_uuid}".
- Each declared property is a nullable typed field (_p_<name>; a datetime is a TIMESTAMPTZ holding its UTC instant, which is all a filter compares; the segment store keeps the offset). Undeclared properties go in one JSON field, where a filter compares one only with values of a comparable type, by its type tag, as it does a declared field: an int compares with an int or a float, a float only with a float.
- Negation is the complement, as on Qdrant.
Index:
- HNSW_SQ with AUTOINDEX's own parameters (M=18, efConstruction=240, 4-bit codes rescored against FP16), named so that every server builds it.
- A search sets only refine_k = 8. On the 100k-row tenant of a 180k-vector test set (Milvus 2.6.24), recall@10 is 0.988-0.990, against 0.73 with no search parameters, at 2.2-2.3 ms per search.
- Scores are the server's.
- Cosine, inner product, and Euclidean are the metrics; another is refused before the registry reserves the name.
Client: AsyncMilvusClient, with every request bounded by MilvusConf.request_timeout_seconds. Under the synchronous client on asyncio.to_thread, with 32 Strong gets in flight beside 8 searchers, slow reads exhausted the shared executor: 12-21 searches per second, against 670-752 on the async client.
Consistency: reads run at Bounded, the level the store names when it creates a native collection, at most the server's common.gracefulTime behind. Session had stalled every search behind the same process's writes (p99 5.3-5.8 s, against 39-58 ms at Bounded). MilvusConf.consistency_level goes.
Upsert: a batch Milvus refuses as over its request size limit (proxy.grpc.serverMaxRecvSize, gRPC's RESOURCE_EXHAUSTED status) is halved until its halves fit or a single entity is refused, as the Qdrant store does; any other error raises at once. The milvus extra declares grpcio for the status.
Purge: a round lists and deletes up to MilvusConf.purge_batch_size (10,000) primary keys. A delete raises unless Milvus accepted every key sent. A round first indexes and loads a native collection that a failed creation left part-made, since Milvus answers the listing query only on a loaded collection. That step is a deliberate stopgap: it adds about 2 ms to a round (see Verification), and [vector store scale-out 6/6] Make a vector store one collection, with string-keyed partitions #1627 removes it. There the store creates, indexes, and loads its one native collection in startup, so no purge round meets a part-made one.
Configuration:
- MilvusConf.tombstone_retention_seconds has the same floor as Qdrant's.
- MilvusConf.max_varchar_length is a declared string's VARCHAR length; a native collection keeps the length it was created with.
- Limits the server configures stay the server's, among them common.JSONMaxLength on a record's undeclared properties. A declared property's field name, its key of at most 32 bytes under _p_, fits proxy.maxNameLength (255 unless configured).
- MilvusConf.metrics_factory_id names the metrics factory the store reports its operation latencies to, as Qdrant's does.
Milvus Lite is dropped. It is a separate embedded engine that scores, indexes and enforces collection properties differently.
- MilvusConf.uri defaults to http://localhost:19530, and a URI ending in .db, which pymilvus serves with Milvus Lite, is refused.
- The milvus extra no longer installs milvus-lite.
- The supported servers are Milvus 2.6.8 and later.
Tests: the store's tests, with the collection lifecycle contract, run against a Milvus 2.6.24 server container as integration tests. They pin query and delete isolation between collections sharing a native collection; ranking, scores and the threshold for every metric; reads at Bounded or stronger; random filters against the in-memory evaluator and a seeded operation sequence against a model; purge rounds on a dropped collection and with two purgers; concurrent preparation of one native collection; non-default settings reaching the client; a match scoring exactly the threshold kept; filters on characters outside the Basic Multilingual Plane and on undeclared properties of another type; a purge round completing a part-made native collection; a declared datetime whose offset has a seconds component; an upsert over the request size limit; and churn across two stores sharing one registry, with purgers running. The test containers need testcontainers 4.15.0.

Breaking, with no migration:

Existing Milvus data is not carried over: drop the Milvus collections before upgrading. The native collection keeps its name, and the store cannot prepare an existing collection of the earlier schema, so the first request of every new session would fail.
Every Milvus configuration must name a collection_registry.
Milvus Lite URIs are refused.

Commits

Move the Milvus store onto the collection registry, against a Milvus server.
Document how the Milvus store meets the shared contracts.
Build Milvus handles from live registrations.
Take a Registration in the Milvus handle.
Say to drop the Milvus collections before upgrading.
Test that Milvus queries and deletes stay within their collection.
Test the Milvus score threshold and ranking for every metric.
Test that the Milvus store reads at Bounded or stronger.
Check random Milvus filters against the in-memory evaluator.
Check a random sequence of Milvus operations against a model.
Test Milvus purge rounds on a dropped collection and on two purgers.
Test concurrent preparations of one Milvus native collection.
Test Milvus settings with values other than their defaults.
Test concurrent churn across two Milvus stores with purgers running.
Give the Milvus wiring tests' collection registry a file database.
Test that the Milvus store keeps a match scoring exactly the threshold.
Use serial commas in the Milvus document and the store's comments.
Name the Milvus filter helpers for what they produce.
Send Milvus filter strings as UTF-8, not ASCII escapes.
Compare an undeclared Milvus property only with values of a comparable type.
Index and load a part-made Milvus collection before a purge round lists it.
Drop the Milvus store's already-exists guard, which no supported server reaches.
Refuse a metric Milvus does not support before reserving the collection.
Bound the Milvus connectivity check by the request timeout.
Require testcontainers 4.15.0, the first with the community modules.
Dispose a mocked-client purge test's registry engine when it fails.
Drop the Milvus store's re-sort of search results.
Say that a Milvus native collection keeps its VARCHAR length.
Match no declared Milvus int property with a float filter value.
Create Milvus native collections at Bounded by name.
Report Milvus store operation latencies to a metrics factory.
Name Milvus's cap on a record's undeclared properties.
Refuse a Milvus Lite URI by its .db suffix, as pymilvus selects Lite.
Write a declared Milvus datetime as its UTC instant.
Halve a Milvus upsert refused as too large, as the Qdrant store does.
Name Milvus's datetime offset fields for their unit.
Keep a declared Milvus datetime as its UTC instant alone.

Stack

21 open PRs: three independent PRs, and the vector store tree of short parallel branches. Every PR in the tree has feat/horizontal-scaling as its GitHub base, and the independent PRs have main. The branches are in a fork, and a pull request can target only this repository's branches, so the on column gives the order the PRs build on each other. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

#	PR	change	on
—	#1624	Make no memory request create a project	`main`
—	#1786	Refuse a filter value of the wrong type for a datetime column, and answer an invalid list filter with 422 (port of #1620)	`main`
—	#1792	Refuse property values that some store refuses or alters where an episode enters	`main`

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1670 and #1702 sit beneath #1627, whose code depends on them. Until the PRs under it merge, their changes show in a stacked PR's diff.

#	PR	change	on
[vector store scale-out 1/6]	#1671 (merged)	Remove custom sharding from the Qdrant store (port of #1654)	`main`
[vector store scale-out 2/6]	#1733 (merged into `feat/horizontal-scaling`)	Answer vector store queries with record UUIDs and scores, and refuse invalid inputs	`feat/horizontal-scaling`
[vector store scale-out 3/6]	#1734 (merged into `feat/horizontal-scaling`)	Arbitrate vector store collections in a SQL registry, with an incarnation per collection life and a purge	`feat/horizontal-scaling`
[vector store scale-out 4/6]	#1735 (merged into `feat/horizontal-scaling`)	Move the Qdrant store onto the collection registry	`feat/horizontal-scaling`
[vector store scale-out 5/6]	#1736 (this PR)	Move the Milvus store onto the collection registry, against a Milvus server	`feat/horizontal-scaling`
—	#1775 (merged into `feat/horizontal-scaling`)	Accept attempts exhausted in the lifecycle churn contract, and say which Qdrant operations filter on the incarnation	`feat/horizontal-scaling`
—	#1779 (merged into `feat/horizontal-scaling`)	Classify Qdrant errors by status code alone	`feat/horizontal-scaling`
—	#1788	Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract	#1736
—	#1631 (closed)	Superseded by #1733–#1736, which hold its changes split in four, with review changes since	—
[user properties 1/2]	#1670	Remove per-project filterable properties (port of #1606)	#1736
[user properties 2/2]	#1702	Keep user properties out of the vector store	#1670
[vector store scale-out 6/6]	#1627	Make a vector store one collection, with string-keyed partitions	#1702
[session storage 1/2]	#1622	Create a session's storage with the session, never on a request	#1627
[session storage 2/2]	#1625	Remove open-or-create from both stores, and close from the segment store	#1622
[search results]	#1663	Score every vector search by cosine similarity, and name scores for it (port of #1598's cosine half)	#1627
[sqlite store fixes 1/7]	#1460	Publish vector index files atomically (but not durably)	#1663
[sqlite store fixes 2/7]	#1469	Never reuse a row id in SQLiteVectorStore	#1460
[sqlite store fixes 3/7]	#1672	Own the search engine's concurrency in the store, not in each engine (port of #1612)	#1469
[sqlite store fixes 4/7]	#1673	Serialize a partition's writes so the engine sees them in order (port of #1607)	#1672
[sqlite store fixes 5/7]	#1674	Refuse a pending row replay cannot honor, instead of dropping it (port of #1608)	#1673
[sqlite store fixes 6/7]	#1675	Take SQLite's write lock at BEGIN, not at the first write (port of #1609)	#1674
[sqlite store fixes 7/7]	#1676	Give every write a fresh row id, so a key names one version (port of #1610)	#1675
[qdrant options]	#1618	Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization	#1663
[declared schema 1/2]	#1628	Make a vector store filter only on the properties it declares	#1663
[declared schema 2/2]	#1616	Close the filter union, and make negation the complement on every backend	#1628

This PR is its 37 commits, aa809bd91, 955f84a89, 5582cf4d6, 8e6539180, caefa7b73, 509c2359d, 7f1eb3b42, 6e312e078, 6d1ed1e92, 25e79487d, 2575af03c, 6f368777f, d43e9e611, 6dfbf595a, 1b94f7f9f, cc29d2765, 3681d092e, bc6d4f9cd, 1a7fffa42, d6fc2199b, 28e8c507f, 9fd7f76d5, b3ef4f1ef, 76062f81d, 6dd2f9c87, 6fcd8c5c2, c264b3d94, 9696e6e2c, 31fc7c465, baabf4899, 07e6ce899, dac193a56, 25fab8f2d, ca890c8b1, f56686f28, b01522967, 49b9ec2dd, directly on feat/horizontal-scaling. #1788 and #1670 are stacked on it, in parallel. Split from #1631 (closed).

Verification

Commit 37: at the head 49b9ec2dd: ruff check, ruff format --check, and ty check are clean, and uv lock --check passes; the full server suite without integration tests passes (2169 passed, 8 skipped); in test containers, the integration tests pass (1676 passed, 266 skipped). The schema test asserts the native collection's exact field set, with no offset field, and the datetime tests read back the written instant in UTC.

Commit 36: at the head b01522967: ruff check, ruff format --check, and ty check are clean, and uv lock --check passes; the full server suite without integration tests passes (2169 passed, 8 skipped); in test containers, the integration tests pass (1676 passed, 266 skipped). The new test declares a datetime under a 32-byte key, the longest a property may have, so its fields are named _p_ and _tz_offset_seconds_ plus 32 characters, and Milvus 2.6.24 creates, writes, reads back, and filters on both.

Commit 35: at the head f56686f28: ruff check, ruff format --check, and ty check are clean, and uv lock --check passes; the full server suite without integration tests passes (2169 passed, 8 skipped); in test containers, the integration tests pass (1675 passed, 266 skipped). Against Milvus 2.6.24, a 72 MB upsert is refused in 0.2 s with gRPC's RESOURCE_EXHAUSTED ("received message larger than max (72055171 vs. 67108864)"), raised by pymilvus as it is, with nothing stored; a 32 MB one is accepted. The new integration test fails with RESOURCE_EXHAUSTED with the halving removed, and so does the mocked halving test.

Commits 33-34, after another review of this PR: at the head ca890c8b1: ruff check, ruff format --check, and ty check are clean, and uv lock --check passes; the full server suite without integration tests passes (2165 passed, 8 skipped); in test containers, the integration tests pass (1674 passed, 266 skipped). Commit 33's tests of a file:// URI ending in .db and of a unix: URI fail under the earlier check, which refused a URI without ://. Commit 34's test fails with the datetime written at its own offset: Milvus 2.6.24 refuses +00:19:32 with "invalid timezone name" (code 1100).

Cost of the purge round's index-and-load, measured on this PR's purge and collection-preparation code: against Milvus 2.6.24 in a test container on one machine, with 768-dimension vectors, a collection of 20,000 records purged in rounds of 5,000 beside another collection of 20,000 in the same native collection, three passes with the step and three with it replaced by a no-op. The step alone takes 2.4 ms median (p90 4.5 ms) on a loaded collection. A round that deletes a full batch takes 32 ms median both with and without it, and whole purges took 228-383 ms in either mode.

Commits 30-32, after two more reviews of this PR: at the head dac193a56: ruff check, ruff format --check, and ty check are clean, and uv lock --check passes; the full server suite without integration tests passes (2160 passed, 8 skipped); in test containers, the integration tests pass (1673 passed, 266 skipped), both measured at a head whose tree differs only in how one configuration test builds MilvusConf, which passes at this head. Commit 30's test fails with the level left out of the create request; commit 31's fails with the factory not passed. Commit 32 cites Milvus v2.6.24's common.JSONMaxLength (pkg/util/paramtable/component_param.go, default 64 KiB) and the upsert check that enforces it (internal/proxy/validate_util.go).

Commit 29: at the head 31fc7c465: ruff check, ruff format --check, and ty check are clean, and uv lock --check passes; the full server suite without integration tests passes (2160 passed, 8 skipped); in test containers, the integration tests pass (1673 passed, 266 skipped). Its test fails with a float counted as comparable with an int property: Milvus 2.6.24 refuses the query ("cannot cast value to Int64", code 1100).

Rebased onto feat/horizontal-scaling after it was rebased onto main (2d471c50a): every commit is patch-identical to the one before the rebase; commit 28 changes documentation only. At the head 9696e6e2c: ruff check, ruff format --check, and ty check are clean, and uv lock --check passes; the full server suite without integration tests passes (2160 passed, 8 skipped); in test containers, the integration tests pass (1672 passed, 266 skipped).

Commits 19-27, after a review of this PR: at the head, ruff check, ruff format --check, and ty check are clean, and uv lock --check passes; the full server suite without integration tests passes (2055); in test containers, the integration tests pass (1533 passed, 266 skipped). Each new test failed under a named mutation of the code, listed in its commit message. Against Milvus 2.6.24: a filter string escaped to a surrogate pair fails to parse; a native collection left unloaded refuses the purge's listing with "collection not loaded"; a create of an existing collection with another schema is refused as "create duplicate collection with different parameters", which the removed guard's text did not match; and 450 of 450 result lists came back best first, across the three metrics, filtered and not.

Rebased onto feat/horizontal-scaling after #1734 and #1735 merged there as 5a846570a and f4cb7bd84: every commit's tree is unchanged, so the results below hold.

Commits 6-18, rebased with the stack onto feat/horizontal-scaling: at the head, ruff check, ruff format --check and ty check are clean; the full server suite without integration tests passes (2055); in test containers, the integration tests pass against PostgreSQL, Qdrant 1.19.1 and Milvus 2.6.24 (662 passed, 6 skipped). Each new or changed test failed under a named mutation of the code, listed in its commit message. Milvus 2.6.24 answered identical concurrent creates of one collection with success every time, so the store's already-exists branch is tested with a mocked client.

After the fourth review round of #1733 and #1734: the commits are rebased onto #1735's head a0891bec2, patch-identical to the ones verified at 0022fed7d, with commit 1's conflict in milvus_vector_store.py resolving to the commit's own file. At the head: ruff check, ruff format --check and ty check are clean; the full server suite without integration tests passes (2107); in test containers, the integration tests pass against PostgreSQL, Qdrant 1.19.1 and Milvus 2.6.24 (597 passed, 6 skipped).

After the third review round of #1734: commits 1-4 are patch-identical to the ones verified below, rebased onto #1735's head 0c4f87037, except commit 1, whose conflict in milvus_vector_store.py resolves to the commit's own rewritten file. Commit 5 adds documentation only. At commit 5, as verified before #1734's last test-only change: ruff check, ruff format --check and ty check are clean; the full server suite without integration tests passes (2100); in test containers, the integration tests pass against PostgreSQL, Qdrant 1.19.1 and Milvus 2.6.24 (597 passed, 6 skipped).

After the second review round of #1733-#1735: commits 1-4 are patch-identical to the ones verified below, rebased onto #1735's head d823255de. At commit 4: ruff check, ruff format --check and ty check are clean; the full server suite without integration tests passes (2097); in test containers, the integration tests pass against PostgreSQL, Qdrant 1.19.1 and Milvus 2.6.24 (597 passed, 6 skipped).

Rebased onto #1735. Commits 1 and 2 are patch-identical to the ones verified below, and do not type-check on this base until commit 3, which adapts the Milvus handle and its test to the registration API. At commit 3: ruff check, ruff format --check and ty check are clean; the full server suite without integration tests passes (2085); in test containers, the integration tests pass against PostgreSQL, Qdrant 1.19.1 and Milvus 2.6.24 (590 passed, 6 skipped).

After #1734's rename: commits 1-3 are patch-identical to the ones above, rebased onto #1735's head, and do not type-check there until commit 4, which renames the Milvus handle's uses. At commit 4: ruff check, ruff format --check and ty check are clean; the full server suite without integration tests passes (2085); in test containers, the integration tests pass against PostgreSQL, Qdrant 1.19.1 and Milvus 2.6.24 (590 passed, 6 skipped).

Before the rebase:

At each commit:

ruff check, ruff format --check and ty check are clean, run as CI runs them.
The full server suite without integration tests passes at commit 1 (2083 tests); the Milvus store's tests are integration tests from here on.
Commit 2 adds the design document only.

At the head, in test containers: the integration tests pass against PostgreSQL, Qdrant 1.19.1 and Milvus 2.6.24 (589 passed, 6 skipped). They cover the vector store (the Milvus store with the collection lifecycle contract), the vector-store semantic storage, event memory, long-term memory and the resource manager. The 6 skipped are three Neo4j-specific semantic storage cases, which skip on the other backends, and three long-term memory tests, for want of a NebulaGraph server.

Before the review changes of #1733-#1736, the head's tree was byte-identical to #1631's head 89ba59636 (tree 8f3ad14bc); it now differs from it by those changes.

🤖 Generated with Claude Code

The declared-schema contract gains a test that every store matches strings holding accented letters, an emoji, quotes, a backslash, and control characters, by equality and by membership. It fails on Milvus with `ensure_ascii=False` removed from `_expression_string_literal`, the UTF-8 literal [vector store scale-out 6/6] ports from MemMachine#1736's 1a7fffa. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…er, live from resolve The registry is addressed by partition key only. `register(key, schema)` answers a PendingRegistration, whose `mark_live()` answers the same life as a LiveRegistration or raises VectorStorePartitionDeletedError when the partition was deleted meanwhile, and whose `unregister()` abandons that life alone. `resolve(key)` answers the LiveRegistration of the live partition, None when there is none, and raises VectorStorePartitionPendingError, which now carries the pending partition's schema, when it is pending. A LiveRegistration's `require_current()` raises VectorStorePartitionHandleStaleError once its partition is deleted. `get`, `mark_live(incarnation)`, `unregister_incarnation` and RegisteredPartition are gone, so deletion is by key or by a pending registration, never through a live one. The SQLAlchemy registry supplies frozen-dataclass registrations holding the engine and the vector store name, and a module function runs the unregistration transaction. The base handle takes its live registration, fences on `require_current()`, and `_partition_handle` builds a handle from one. `create_partition` lets VectorStorePartitionDeletedError propagate, and the VectorStore interface names it; open-or-create catches it and creates again, refuses a pending partition of another schema at once from the pending error's schema, and re-raises the last pending error. A `get_partition` of a pending partition of another schema still reports the mismatch first. The Qdrant and Milvus handles take the registration in place of the key, the incarnation and the registry lookup. Tests follow: the registry's tests answer registrations and gain one for `require_current`; the base's tests patch the pending registration type's `unregister` and expect the deleted error from a creation a deletion undid; the lifecycle contract patches the live registration type's `require_current`, registers its racing winners through pending registrations, and counts the deleted error among churn's outcomes; the Qdrant and Milvus tests that build a handle on a mocked client give it a registration that stays current. The same change as MemMachine#1734's 735c649, f88aaa8, e25be60 and b2bb2d8 and the handle halves of MemMachine#1735's 9096edd and MemMachine#1736's 5582cf4, for this PR's partition registry. The event-backend locator change has no counterpart: the locator here opens its partition with open-or-create, which creates again itself. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…nd their partitions The design documents came from MemMachine#1733-MemMachine#1736, where a store holds logical collections addressed by namespace and name, each with its own configuration. Here a store is one collection, with its dimensions, metric and declared schema fixed at construction, and a partition is one tenant's records in it, addressed by key. The documents say so: - the collection registry document becomes the partition registry document: a registry belongs to one store and is addressed by partition key; its tables are `partition_registry_pt` and `partition_registry_gc`, and a tombstone needs no location, since the incarnation alone finds a dead partition's records in the store's native collection; a partition created under another schema is refused; `startup` creates the tables, as every store's startup creates its durable resources; a decision records that a store is one collection; - the Qdrant and Milvus documents lay out one native collection per store, named by the vector store name (`sys_` and the name on Milvus), created at startup, where a partition's creation makes nothing in the backend; - the overview, isolation, consistency and purge documents speak of partitions, `purge_deleted_partitions` and `settle(partition)`, and the overview describes the store and its partitions. The measurements and their conditions are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…gistration PendingRegistration and LiveRegistration named the partition's state when the registry answered them, which a deletion by anyone falsifies: a "live" registration may have been deleted since. What does not change is what each handle's holder may do, so the handles are named for that, in the words of two known patterns: Try-Confirm/Cancel for the creator, and the stale handle for everyone else. - `register` is `reserve`, and answers a `Reservation`: the creator's hold on the key while it prepares the partition's storage. - `PendingRegistration.mark_live` is `Reservation.confirm`, which marks the partition live and answers its `Registration`. - `PendingRegistration.unregister` is `Reservation.cancel`. - `LiveRegistration` is `Registration`; `resolve`, `require_current` and `unregister(partition_key)` keep their names. - The fields both share sit in a private base, `_RegistryEntry`. "Pending" stays the word for the partition's state, in the store contract and VectorStorePartitionPendingError. The base store's creation flow, its task set and log text, the Qdrant and Milvus handles, the tests and the design documents follow; the registry design records why the names are roles. The same change as MemMachine#1734's ffa954c, MemMachine#1735's 87edbad and MemMachine#1736's 8e65391, for this PR's partition registry. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…1734's other review changes MemMachine#1734's latest review changes, for this PR's partition registry and stores: - `run_purge_round(purge_round)` replaces `claim_purgeable_incarnation()`. The registry claims the oldest due tombstone, calls the round with its incarnation inside the claim's transaction, and records what the round returns. `PurgeClaim` and its `any_records_found` guard go; `PurgeRound` takes only the incarnation, since a tombstone here carries nothing else. On SQLite the claim is an `UPDATE ... RETURNING` of the oldest eligible row, so purgers serialize at the claim; PostgreSQL keeps `FOR UPDATE SKIP LOCKED`. A failure count at or past the dead-letter bound is reported. - A reservation's cancel reports its own failure from its task, so a creation cancelled again still has the failure logged. - `get_partition` runs under the tracker like the other lifecycle calls, and the SQLite stores check partition keys with the shared `require_partition_key`. - The registry and purge documents follow. The upgrade notes state what holds for these stores: they name their native collections by vector store name, which no earlier release did, so an existing Qdrant or Milvus collection is never read or purged, and can be dropped before or after upgrading. The Milvus design document's consequence, which said an existing collection has to be dropped, says the same, and the Helm README lists `partition_registry` among the Qdrant store's keys. The same changes as MemMachine#1734's d041785, cd13045, 35b57b7, 0a68f37, d05556b, 118d23b, 5225933 and 783967e, MemMachine#1735's a40bc50 and MemMachine#1736's caefa7b, for this PR's partitions. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…saction across the round A purge round ran inside the claim's transaction: on PostgreSQL a row lock held idle while the backend deleted, and on SQLite the database file's write lock, which every store sharing the file waited on for the whole round. - The claim is a lease. One committed `UPDATE ... RETURNING` takes the oldest due tombstone that no unexpired claim holds, stamping `claimed_at` and incrementing `claim_generation`; the round runs with no transaction open; the writes that end the claim are conditioned on its generation, so a round that outlasted its lease cannot end the claim taken after it. A round that found nothing removes the tombstone under any claim. `purge_lease_seconds` (default 300) sets the lease. - A claim that finds the previous claim unended past its lease runs no round: it counts that round as failed, as of when it was claimed, and logs it. A cancelled round ends its claim uncounted, in a shielded write whose task reports its own failure. - `purge_retry_backoff_seconds` is `base_purge_retry_backoff_seconds`, the first delay the backoff doubles. - The registry refuses an engine on StaticPool or in-memory SQLite, whose connections do not arbitrate as separate transactions; the wiring tests give their registry a file database. The registry tests cover the lease, the outcome writes and their fence, deletion and reservation atomicity under injected faults, two registries sharing a database, collision without waiting, churn across engines, and random operation sequences against a model. The purge and registry design documents describe the lease and the alternatives considered. The same changes as MemMachine#1734's f1e9de8, 054b079, a4e2b24 and 900a926, MemMachine#1735's 3d7761e and 81b1bf5, and MemMachine#1736's 1b94f7f, for this PR's partition registry. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The query contract said nothing of a match scoring exactly the threshold, and Qdrant dropped it: the server compares its threshold in single precision and keeps only scores strictly better than it. The contract now says a match scoring exactly the threshold is returned. The Qdrant store sends the server the adjacent single-precision value on the worse side, and applies the caller's threshold exactly itself; a threshold beyond single precision is not sent. Both SQLite stores already keep the match, and each now tests it on every metric it supports. The same changes as MemMachine#1735's ccba117 and 6bbc131, for this PR's stores. MemMachine#1736's cc29d27, Milvus's test of the same, is ported with the store tests. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…cycle contract's drain MemMachine#1735's and MemMachine#1736's latest tests, for this PR's partitions: - Every Qdrant store test runs against a Qdrant server, over REST and gRPC: local mode ignores payload indexes and raises its own exceptions where a server answers not-found or already-exists, so it is no longer a fixture. The tests on a mocked client stay in the default suite. - Both stores gain tests that partitions and stores of different names keep their records apart, that purge rounds reclaim a write landing after a round and drain two stores' tombstones alone, that ranking, scores and the threshold follow every similarity metric, that a point or entity the server refuses on its own raises, that concurrent creations and startups agree, that two stores churning one registry keep every partition exact, and that seeded operation sequences, and on Milvus random filters, agree with a model. The Milvus tests also pin its read consistency and its settings with values other than their defaults. - The lifecycle contract's drain fails after a bounded number of rounds, it settles before checking that a new life is empty, and it checks a stale upsert by what the partition holds rather than by its registry reads. The same changes as MemMachine#1735's 30c30f4, e9e71b5, 6c2362b, 4d3241c, 836db21, 252bcd2, 5e8c049, ef8d3ab, 276a034, 4fb5bd4 and 2aabd3d, and MemMachine#1736's 509c235, 7f1eb3b, 6e312e0, 6d1ed1e, 25e7948, 6f36877, d43e9e6, 6dfbf59 and cc29d27, for this PR's partitions. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The base store's purge contract says a round that finds the incarnation's storage missing returns False, but neither store's round checked: once its native collection was dropped from outside, every round on it raised, counted against the tombstone, and dead-lettered it after ten. A Qdrant round that the server answers not found, over REST or gRPC, and a Milvus round that finds no native collection, now return False: the collection is gone with everything in it, so the tombstone is retired. Each store has a test that drops its native collection and drains the purge. The same handling as MemMachine#1735's and MemMachine#1736's stores, whose rounds already checked for a missing native collection; MemMachine#1736's 2575af0 tests it. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…n, and use serial commas MemMachine#1734's, MemMachine#1735's, and MemMachine#1736's latest round, for this PR's partition registry and stores; no behavior changes: - The queue's `failed_rounds` column is `consecutive_failed_rounds`, in the code, the tests, the documents, and the dead-letter log's hint. - The registry keeps its durations as seconds. `_has_elapsed(seconds, since=)` answers whether a duration has passed on the database clock, for the retention, the backoff, and the lease, and `_purge_retry_backoff_seconds()` computes each tombstone's capped backoff. - `run_purge_round` runs named steps: `_claim_oldest_due_tombstone` answers a `_TombstoneClaim`, an `_UnendedPurgeRound`, or None, and the round's outcome goes to `_count_failed_purge_round`, `_end_tombstone_claim_after_cancellation`, or `_record_purge_round`. `_insert` is `_insert_pending_partition`, `_claim_releases` `_tombstone_claim_endings`, and the base store's `_cancellations` `_reservation_cancellations`. - The Milvus filter helpers are named for what they produce, comments are shorter, and lists in the documents, comments, and docstrings take a serial comma. The same changes as MemMachine#1734's 2a3d87c and 503687c, MemMachine#1735's 106e106, and MemMachine#1736's 3681d09 and bc6d4f9, for this PR's partitions. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

MemMachine#1736's review round, for this PR's Milvus store: - Filter strings reach Milvus as UTF-8, so a character outside the Basic Multilingual Plane parses, and an undeclared property's condition requires the stored type tag to be one the value compares with. - Startup's already-exists guard goes, with its mocked-client test: Milvus answers a create of an existing collection with the same schema with success. - The store no longer re-sorts search results, which Milvus returns best first. - The mocked-client purge test disposes its registry engine when it fails. The same changes as MemMachine#1736's 1a7fffa, d6fc219, 9fd7f76, 6fcd8c5, and c264b3d, for this PR's store. MemMachine#1736's 28e8c50 and b3ef4f1 have no counterpart here: startup prepares the store's one native collection before any purge round runs, and the store refuses an unsupported metric at construction, before anything is reserved. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The declared path counted an int and a float as comparable either way, so a float filter value on a declared int property reached Milvus as a float literal against an INT64 field, which Milvus refuses to parse ("cannot cast value to Int64", code 1100). A float now compares only with a float property, and matches no int one, as a value of another type matches nothing; an int still compares with a float property by value. The new test filters a declared float property with an int and a declared int property with a float; it fails with the float counted as comparable with an int property. The same change as MemMachine#1736's 31fc7c4, for this PR's store. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…y name MemMachine#1736's baabf48 creates Milvus native collections at Bounded by name, and this PR's base carries that into startup's create, so the read level the purge and the tombstone retention rely on no longer comes from pymilvus's default. The consistency test now spies create_collection and checks the level it names, as MemMachine#1736's test does; it fails with the level dropped from the create. The design document and the store's docstring say the store creates its one native collection at Bounded. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

MemMachine#1736's f56686f halves a Milvus upsert refused with RESOURCE_EXHAUSTED, and this PR's base carries the halving into the partition handle's `_upsert`. Its tests come here in partition terms: the integration test upserts 1,200 records with a 60,000-character property, about 72 MB, and fails with RESOURCE_EXHAUSTED without the halving; the mocked-client tests pin the halving, a single refused entity raising, and a timeout or another refusal sent once, on a handle `_partition_on` builds, which the mocked delete test now uses too. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…nt alone MemMachine#1736's 49b9ec2 drops the offset field beside a declared Milvus datetime, keeping its UTC instant alone, and this PR's base carries that into the store. The tests follow in partition terms: the native collection's fields are exactly the fixed ones and one per declared property; a declared datetime is stored as its instant in UTC; and a store whose declared datetime has the longest property key, 32 bytes, writes its one field, reads it back, and matches the record by its instant, the test MemMachine#1736's b015229 adds. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

A declared datetime on the SQLite stores had a `tz_<key>` column beside its microseconds column, holding the UTC offset in seconds that no filter reads. The vector stores keep a datetime property's instant only, as MemMachine#1736 now does for Milvus and MemMachine#1788 for Qdrant, and the segment store keeps the offset, so the column, `offset_column_name`, and the value written to it go: each declared property is one column, and a datetime stays microseconds since the epoch. The tests read a stored datetime back as its instant in UTC, and the roundtrip test checks that the stored instant equals the written one. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The declared-schema contract gains a test that every store matches strings holding accented letters, an emoji, quotes, a backslash, and control characters, by equality and by membership. It fails on Milvus with `ensure_ascii=False` removed from `_expression_string_literal`, the UTF-8 literal [vector store scale-out 6/6] ports from MemMachine#1736's 1a7fffa. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

edwinyyyu force-pushed the feat/vector-store-milvus-registry-speedkick branch from 973ea51 to 30c1d79 Compare October 1, 2026 22:30

edwinyyyu added the horizontal scaling Wrong or unsafe when more than one server process serves the same backends (replicas or workers) label Oct 1, 2026

edwinyyyu force-pushed the feat/vector-store-milvus-registry-speedkick branch from 30c1d79 to 2f72832 Compare October 1, 2026 23:51

edwinyyyu mentioned this pull request Oct 2, 2026

[event memory 1/7] Add time bounds, sources and expansion to EventMemory (port of #1597, without session) #1684

Draft

edwinyyyu force-pushed the feat/vector-store-milvus-registry-speedkick branch from 2f72832 to 196d45e Compare October 2, 2026 03:19

edwinyyyu marked this pull request as ready for review October 2, 2026 17:08

edwinyyyu mentioned this pull request Oct 2, 2026

[milvus options] Let a deployment tune a Milvus collection's vector index and its searches #1741

Draft

edwinyyyu requested review from malatewang and marvinyu-memverge October 2, 2026 20:14

edwinyyyu merged commit cd96676 into MemMachine:feat/horizontal-scaling Oct 9, 2026
42 checks passed

Navigation Menu

Repository navigation

[vector store scale-out 5/6] Move the Milvus store onto the collection registry, against a Milvus server - #1736

[vector store scale-out 5/6] Move the Milvus store onto the collection registry, against a Milvus server#1736
edwinyyyu merged 37 commits into
MemMachine:feat/horizontal-scalingfrom
edwinyyyu:feat/vector-store-milvus-registry-speedkick

edwinyyyu commented Oct 1, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

edwinyyyu commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the change

Summary

Commits

Stack

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

edwinyyyu commented Oct 1, 2026 •

edited

Loading