Skip to content

[declared schema 2/2] Close the filter union, and make negation the complement on every backend - #1616

Draft
edwinyyyu wants to merge 33 commits into
MemMachine:feat/horizontal-scalingfrom
edwinyyyu:feat/filter-union-speedkick
Draft

edwinyyyu wants to merge 33 commits into
MemMachine:feat/horizontal-scalingfrom
edwinyyyu:feat/filter-union-speedkick

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the change

The filter language becomes a closed union: Equals, Ordering, In, IsNull, n-ary And/Or, Not (common/filter/filter_expression.py); every compiler is an exhaustive match. != and <> parse as Not(Equals), NOT IN as Not(In), IS NOT NULL as Not(IsNull), so a negation means one thing. An ordering comparison takes a number or a date: field > 'string' and field > true, which parsed before, are now rejected by the parser.

A leaf on a field holding no value is false, IsNull on it is true, and Not is the complement over the record set on every vector store and the segment store. The SQL compilers render it as NOT COALESCE(x, FALSE): sql_filter_util.py compiled ~inner, three-valued, and a negated filter silently dropped rows holding no value on every SQL-backed store. Milvus already pushes Not to the leaves by De Morgan, since [vector store scale-out 5/6] (#1736), as its own not excludes entities lacking the field, and Qdrant's must_not already admits them. The Neo4j and NebulaGraph graph stores and the Neo4j semantic storage keep their NOT over a comparison three-valued. A filter is bound to the store's declared schema before the store compiles it (bind_filter), and a record before the store writes it (bind_record): an int for a float key becomes the float it equals, and a float for an int key, a bool for a number, a non-finite value, or an int outside 64 signed bits raises PropertyTypeMismatchError, so no compiler meets a value of another type than its key declares. An In over no values admits nothing: every compiler renders it as false, so EventMemory's empty id lists keep meaning "nothing", and its complement admits everything.

Last in the stack, because the store work is the priority and this change touches many files mechanically; the slices under it are written against the nodes main has today. The complement law is checked over thirteen filters in declared_schema_contract.py, across the column, JSON and properties encodings of the SQL stores.

Adaptation for main: the routing of [user properties 1/2] is rewritten on the closed union (And.operands; Equals for Comparison with =); the speedkick-era commit 8ca5f06 was written on #1597's tree, and its rewrites of #1597's _system_predicates, _lift_typed_filters and their tests have nothing to apply to on main. The event memory test spells the field context.producer, main's name for it. The tests the rebuilt base adds use the closed union too, and the segment store's random context-read test models the new semantics: a condition on a property with no value is false and a negation is the complement, where the model followed SQL's three-valued logic. Same 60 files as the origin.

Position: on [search results] (#1663), which sits on [declared schema 1/2] (#1628), in parallel with the SQLite fixes; the fixes' tests, which the origin also converts to the closed union, are not in this branch.

Fixes #1780 and #1781.

Stack

21 open PRs: three independent PRs, and the vector store tree of short parallel branches. Every PR in the tree has feat/horizontal-scaling as its GitHub base, and the independent PRs have main. The branches are in a fork, and a pull request can target only this repository's branches, so the on column gives the order the PRs build on each other. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

# PR change on
— #1624 Make no memory request create a project main
— #1786 Refuse a filter value of the wrong type for a datetime column, and answer an invalid list filter with 422 (port of #1620) main
— #1792 Refuse property values that some store refuses or alters where an episode enters main

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1702 and #1670 sit beneath #1627, whose code depends on them. Until the PRs under it merge, their changes show in a stacked PR's diff.

# PR change on
[vector store scale-out 1/6] #1671 (merged) Remove custom sharding from the Qdrant store (port of #1654) main
[vector store scale-out 2/6] #1733 (merged into feat/horizontal-scaling) Answer vector store queries with record UUIDs and scores, and refuse invalid inputs feat/horizontal-scaling
[vector store scale-out 3/6] #1734 (merged into feat/horizontal-scaling) Arbitrate vector store collections in a SQL registry, with an incarnation per collection life and a purge feat/horizontal-scaling
[vector store scale-out 4/6] #1735 (merged into feat/horizontal-scaling) Move the Qdrant store onto the collection registry feat/horizontal-scaling
[vector store scale-out 5/6] #1736 (merged into feat/horizontal-scaling) Move the Milvus store onto the collection registry, against a Milvus server feat/horizontal-scaling
— #1775 (merged into feat/horizontal-scaling) Accept attempts exhausted in the lifecycle churn contract, and say which Qdrant operations filter on the incarnation feat/horizontal-scaling
— #1779 (merged into feat/horizontal-scaling) Classify Qdrant errors by status code alone feat/horizontal-scaling
— #1813 (merged into feat/horizontal-scaling) Create Qdrant collections with strict mode off feat/horizontal-scaling
— #1788 Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract feat/horizontal-scaling
— #1631 (closed) Superseded by #1733–#1736, which hold its changes split in four, with review changes since —
[user properties 1/2] #1702 Keep undeclared properties out of the vector store feat/horizontal-scaling
[user properties 2/2] #1670 Remove per-project filterable properties (port of #1606) #1702
[vector store scale-out 6/6] #1627 Make a vector store one collection, with string-keyed partitions #1670
[session storage 1/2] #1622 Create a session's storage with the session, never on a request #1627
[session storage 2/2] #1625 Remove open-or-create from both stores, and close from the segment store #1622
[declared schema 1/2] #1628 Make a vector store filter only on the properties it declares #1627
[search results] #1663 Score every vector search by cosine similarity, and name scores for it (port of #1598's cosine half) #1628
[declared schema 2/2] #1616 (this PR) Close the filter union, and make negation the complement on every backend #1663
[sqlite store fixes 1/7] #1460 Publish vector index files atomically (but not durably) #1663
[sqlite store fixes 2/7] #1469 Never reuse a row id in SQLiteVectorStore #1460
[sqlite store fixes 3/7] #1672 Own the search engine's concurrency in the store, not in each engine (port of #1612) #1469
[sqlite store fixes 4/7] #1673 Serialize a partition's writes so the engine sees them in order (port of #1607) #1672
[sqlite store fixes 5/7] #1674 Refuse a pending row replay cannot honor, instead of dropping it (port of #1608) #1673
[sqlite store fixes 6/7] #1675 Take SQLite's write lock at BEGIN, not at the first write (port of #1609) #1674
[sqlite store fixes 7/7] #1676 Give every write a fresh row id, so a key names one version (port of #1610) #1675
[qdrant options] #1618 Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization #1663
[milvus options] #1741 Let a deployment tune a Milvus collection's vector index and its searches #1618

This PR is its 5 commits, 5a7123ccb, 94ae40cfc, 1c4e95b65, 3a3726a23, 13d104245, stacked on #1663.

Verification

At this PR's head 13d104245, on 2026-10-09, on feat/horizontal-scaling with #1813 merged: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean; the Qdrant store's, the registry-backed store's, and the database manager's tests, unit and integration, pass, 386 tests, against Qdrant 1.19.1 defaulting new collections to strict mode and Milvus 2.6.24 in containers.

Before the rebase onto #1813's merge: At this PR's head df21dcdda, on 2026-10-09: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. Its tree is the previous head e9b2fc25c's: the server suite without integration tests last passed at d491433ce, on 2026-10-08, 2235 tests, and the integration tests of the vector stores, the resource manager, episodic memory, and semantic storage at d0ba0b451, on 2026-10-08, 840 tests, against PostgreSQL 16, Neo4j, Qdrant 1.19.1, and Milvus 2.6.24 in containers. The SQLite stores' tests last passed at 372d94164, on 2026-10-09, 270 tests, after the SQLite offset column was dropped.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

@edwinyyyu
edwinyyyu marked this pull request as draft September 14, 2026 19:06
@edwinyyyu
edwinyyyu force-pushed the feat/filter-union-speedkick branch 5 times, most recently from e5f728a to bdadfe0 Compare September 14, 2026 20:33
@edwinyyyu
edwinyyyu force-pushed the feat/filter-union-speedkick branch from bdadfe0 to f8ca110 Compare September 14, 2026 21:42
@edwinyyyu edwinyyyu changed the title (Depends on #1606) Close the filter union, and make negation the complement on every backend (speedkick) [vector store 13/13] Close the filter union, and make negation the complement on every backend (speedkick) Sep 14, 2026
@edwinyyyu
edwinyyyu force-pushed the feat/filter-union-speedkick branch 5 times, most recently from 219905a to 792c4e9 Compare September 14, 2026 23:16
@edwinyyyu edwinyyyu changed the title [vector store 13/13] Close the filter union, and make negation the complement on every backend (speedkick) [vector store 12/12] Close the filter union, and make negation the complement on every backend (speedkick) Sep 14, 2026
@edwinyyyu
edwinyyyu force-pushed the feat/filter-union-speedkick branch 2 times, most recently from 219905a to ed99d4b Compare September 15, 2026 17:26
edwinyyyu and others added 30 commits October 9, 2026 17:04
…uilt by the composition root

A vector store was a factory of logical collections, each identified by
a (namespace, name) pair and created with its own dimensions, metric and
schema; Qdrant and Milvus shared one native collection among logical
collections of equal configuration, under a name derived from a hash of
that configuration, and the registry mapped (namespace, name) to it.

A store is now one collection: `VectorStore(vector_store_name,
vector_dimensions, similarity_metric, indexed_properties)` names its one
native collection (or its tables and index files) at construction, every
partition of it shares the store's dimensions, metric and schema, and
`provision()` creates the store's durable resources idempotently, before
`startup`. `create_partition(key)`, `open_or_create_partition(key)`,
`get_partition(key)` and `delete_partition(key)` take a string key; a
partition is the records carrying its incarnation in the store's native
collection (Qdrant, Milvus) or a pair of tables (the SQLite stores), and
the registry records what each partition was created under, so a store
built with other dimensions, another metric or another schema raises
`VectorStorePartitionSchemaMismatchError` instead of reading columns and
vectors that are not there. "Collection" names only the native Qdrant or
Milvus collection. Vector store names match `[a-z0-9_]+` and are at most
32 bytes, the rule partition and property keys follow, so every name
works on every backend: the Qdrant store names its collection by the
vector store name, and the Milvus store by `sys_` followed by it, since
Milvus requires a leading letter or underscore and names its own
internals with a leading underscore. The hash-derived native names go,
and with them the per-partition config.

The partition registry is keyed by partition key within one store: its
tables, `partition_registry_pt` and `partition_registry_gc`, are shared
by every store on a database and keyed by vector store name, so the
registries of several stores share one database and nothing else.
`mark_live` and `unregister_incarnation` take the incarnation alone. On
the registry-backed base, the storage a store's partitions share is
prepared by `provision()` (`_prepare_storage()`), and the storage a
partition keeps of its own by `_prepare_partition_storage(key,
incarnation)`, between its registration and its mark; the Qdrant and
Milvus stores keep nothing per partition. A purge round works on one
incarnation in the store's own collection.

`DatabaseManager.get_vector_store(backend, vector_store_name=,
vector_dimensions=, similarity_metric=, indexed_properties=)` builds and
caches one store per (backend, vector store name), provisioning its
registry and then the store; asking for a name again with other
dimensions, another metric or other keys is a
`VectorStoreConfigurationError`. The event backend's store is named by a
UUIDv5 of its embedder id in the UUIDv5 of its backend's key in a namespace
fixed for the event backend, written as 32 hexadecimal digits, so neither
the key nor the id is constrained; stores on different backends never share
a name, so their registries stay apart when the backends share a registry
database, since a store name identifies one store among all the stores
whose registries share it.
Semantic memory's one store is named `semantic_memory` whatever its
embedder and holds every org's features in one partition,
`semantic_memory`, as main's one collection does.
The event backend opens a session's partition with
`open_or_create_partition`, which waits on one another worker is still
creating.

The SQLite stores change shape only: their registry tables become
`vector_store_sqlite_pt` and `vector_store_sqlite_vec_pt`, keyed by
vector store name and partition key, the pending-operation log is keyed
the same way, and per-partition table names embed the vector store name.
No migration is provided; existing SQLite vector data is orphaned.

Rebuilt as one change on MemMachine#1631: this PR's earlier history carried copies
of MemMachine#1631's commits as of `38684b385`, its own commits on them, and
mirrors of MemMachine#1631's later commits in its terms. Its code is that history's
final tree, which was verified, merged with main and without MemMachine#1624's
commits. The design documents are still MemMachine#1631's and describe collections.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…up, not a separate provision

A vector store and its partition registry each had a provision() that
created their durable resources, run by the composition root before
startup(). The split between provisioning a store and starting it is
MemMachine#1570's to make for every store at once, so here startup() does both
again, as on main: a store's startup prepares the storage its
partitions share (the native collection and its indexes, or the SQLite
tables), and the registry's startup creates its tables, as MemMachine#1631's does.
The database manager starts the registry, then the store.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…er, live from resolve

The registry is addressed by partition key only. `register(key, schema)`
answers a PendingRegistration, whose `mark_live()` answers the same life
as a LiveRegistration or raises VectorStorePartitionDeletedError when the
partition was deleted meanwhile, and whose `unregister()` abandons that
life alone. `resolve(key)` answers the LiveRegistration of the live
partition, None when there is none, and raises
VectorStorePartitionPendingError, which now carries the pending
partition's schema, when it is pending. A LiveRegistration's
`require_current()` raises VectorStorePartitionHandleStaleError once its
partition is deleted. `get`, `mark_live(incarnation)`,
`unregister_incarnation` and RegisteredPartition are gone, so deletion is
by key or by a pending registration, never through a live one. The
SQLAlchemy registry supplies frozen-dataclass registrations holding the
engine and the vector store name, and a module function runs the
unregistration transaction.

The base handle takes its live registration, fences on
`require_current()`, and `_partition_handle` builds a handle from one.
`create_partition` lets VectorStorePartitionDeletedError propagate, and
the VectorStore interface names it; open-or-create catches it and creates
again, refuses a pending partition of another schema at once from the
pending error's schema, and re-raises the last pending error. A
`get_partition` of a pending partition of another schema still reports the
mismatch first. The Qdrant and Milvus handles take the registration in
place of the key, the incarnation and the registry lookup.

Tests follow: the registry's tests answer registrations and gain one for
`require_current`; the base's tests patch the pending registration type's
`unregister` and expect the deleted error from a creation a deletion
undid; the lifecycle contract patches the live registration type's
`require_current`, registers its racing winners through pending
registrations, and counts the deleted error among churn's outcomes; the
Qdrant and Milvus tests that build a handle on a mocked client give it a
registration that stays current.

The same change as MemMachine#1734's 735c649, f88aaa8, e25be60 and
b2bb2d8 and the handle halves of MemMachine#1735's 9096edd and MemMachine#1736's
5582cf4, for this PR's partition registry. The event-backend locator
change has no counterpart: the locator here opens its partition with
open-or-create, which creates again itself.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…nd their partitions

The design documents came from MemMachine#1733-MemMachine#1736, where a store holds logical
collections addressed by namespace and name, each with its own
configuration. Here a store is one collection, with its dimensions, metric
and declared schema fixed at construction, and a partition is one tenant's
records in it, addressed by key. The documents say so:

- the collection registry document becomes the partition registry
  document: a registry belongs to one store and is addressed by partition
  key; its tables are `partition_registry_pt` and `partition_registry_gc`,
  and a tombstone needs no location, since the incarnation alone finds a
  dead partition's records in the store's native collection; a partition
  created under another schema is refused; `startup` creates the tables, as
  every store's startup creates its durable resources; a decision records
  that a store is one collection;
- the Qdrant and Milvus documents lay out one native collection per store,
  named by the vector store name (`sys_` and the name on Milvus), created at
  startup, where a partition's creation makes nothing in the backend;
- the overview, isolation, consistency and purge documents speak of
  partitions, `purge_deleted_partitions` and `settle(partition)`, and the
  overview describes the store and its partitions.

The measurements and their conditions are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…gistration

PendingRegistration and LiveRegistration named the partition's state when
the registry answered them, which a deletion by anyone falsifies: a "live"
registration may have been deleted since. What does not change is what
each handle's holder may do, so the handles are named for that, in the
words of two known patterns: Try-Confirm/Cancel for the creator, and the
stale handle for everyone else.

- `register` is `reserve`, and answers a `Reservation`: the creator's hold
  on the key while it prepares the partition's storage.
- `PendingRegistration.mark_live` is `Reservation.confirm`, which marks the
  partition live and answers its `Registration`.
- `PendingRegistration.unregister` is `Reservation.cancel`.
- `LiveRegistration` is `Registration`; `resolve`, `require_current` and
  `unregister(partition_key)` keep their names.
- The fields both share sit in a private base, `_RegistryEntry`.

"Pending" stays the word for the partition's state, in the store contract
and VectorStorePartitionPendingError. The base store's creation flow, its
task set and log text, the Qdrant and Milvus handles, the tests and the
design documents follow; the registry design records why the names are
roles.

The same change as MemMachine#1734's ffa954c, MemMachine#1735's 87edbad and MemMachine#1736's
8e65391, for this PR's partition registry.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…ain open-or-create's give-up

Two review changes to MemMachine#1734's registry and base store, for this PR's
partitions:

- `Reservation.cancel()` acts only while the partition is pending, as
  `confirm()` does. A creator whose confirmation committed but whose
  answer was lost, and which then cancels, leaves the live partition
  alone: only a deletion by key ends it. The registry contract, the
  SQLAlchemy registry and the registry design document say so, and a
  test cancels after a confirmation.
- Open-or-create's `VectorStoreAttemptsExhaustedError` is raised from the
  race it last lost, the last `VectorStorePartitionAlreadyExistsError` or
  `VectorStorePartitionDeletedError` it caught. A partition that stays
  pending still raises the pending error itself. A base-store test checks
  the cause.

The same changes as MemMachine#1734's 9daf7e7 and 93a85da, for this PR's
partition registry.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…1734's other review changes

MemMachine#1734's latest review changes, for this PR's partition registry and stores:

- `run_purge_round(purge_round)` replaces `claim_purgeable_incarnation()`.
  The registry claims the oldest due tombstone, calls the round with its
  incarnation inside the claim's transaction, and records what the round
  returns. `PurgeClaim` and its `any_records_found` guard go; `PurgeRound`
  takes only the incarnation, since a tombstone here carries nothing else.
  On SQLite the claim is an `UPDATE ... RETURNING` of the oldest eligible
  row, so purgers serialize at the claim; PostgreSQL keeps `FOR UPDATE SKIP
  LOCKED`. A failure count at or past the dead-letter bound is reported.
- A reservation's cancel reports its own failure from its task, so a
  creation cancelled again still has the failure logged.
- `get_partition` runs under the tracker like the other lifecycle calls,
  and the SQLite stores check partition keys with the shared
  `require_partition_key`.
- The registry and purge documents follow. The upgrade notes state what
  holds for these stores: they name their native collections by vector
  store name, which no earlier release did, so an existing Qdrant or Milvus
  collection is never read or purged, and can be dropped before or after
  upgrading. The Milvus design document's consequence, which said an
  existing collection has to be dropped, says the same, and the Helm
  README lists `partition_registry` among the Qdrant store's keys.

The same changes as MemMachine#1734's d041785, cd13045, 35b57b7, 0a68f37,
d05556b, 118d23b, 5225933 and 783967e, MemMachine#1735's a40bc50 and
MemMachine#1736's caefa7b, for this PR's partitions.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…r is cancelled

A creation cancelled its reservation only when preparing the partition's
storage raised. A confirmation that raised, or a creation cancelled while
it confirmed, left the partition pending until someone deleted it.

The base store's creation step now prepares the storage and confirms the
reservation together, and cancels the reservation if either raises or the
creation is cancelled, shielded as before; the cancel's task reports its
own failure. The cancel acts only on a pending partition, so a
confirmation that committed before its failure was observed stands.
create_partition and open-or-create both go through it; open-or-create
still takes a confirmation's VectorStorePartitionDeletedError as a race to
create again. Base-store tests cover a failed confirmation, a cancelled
one, and one that committed before failing. The registry design document
says so, and the purge document says which writers wait on SQLite's lock
during a purge round.

The same changes as MemMachine#1734's f4c585e and 4dde7c8, for this PR's
partitions.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…saction across the round

A purge round ran inside the claim's transaction: on PostgreSQL a row lock
held idle while the backend deleted, and on SQLite the database file's write
lock, which every store sharing the file waited on for the whole round.

- The claim is a lease. One committed `UPDATE ... RETURNING` takes the
  oldest due tombstone that no unexpired claim holds, stamping `claimed_at`
  and incrementing `claim_generation`; the round runs with no transaction
  open; the writes that end the claim are conditioned on its generation, so
  a round that outlasted its lease cannot end the claim taken after it. A
  round that found nothing removes the tombstone under any claim.
  `purge_lease_seconds` (default 300) sets the lease.
- A claim that finds the previous claim unended past its lease runs no
  round: it counts that round as failed, as of when it was claimed, and
  logs it. A cancelled round ends its claim uncounted, in a shielded write
  whose task reports its own failure.
- `purge_retry_backoff_seconds` is `base_purge_retry_backoff_seconds`, the
  first delay the backoff doubles.
- The registry refuses an engine on StaticPool or in-memory SQLite, whose
  connections do not arbitrate as separate transactions; the wiring tests
  give their registry a file database.

The registry tests cover the lease, the outcome writes and their fence,
deletion and reservation atomicity under injected faults, two registries
sharing a database, collision without waiting, churn across engines, and
random operation sequences against a model. The purge and registry design
documents describe the lease and the alternatives considered.

The same changes as MemMachine#1734's f1e9de8, 054b079, a4e2b24 and 900a926,
MemMachine#1735's 3d7761e and 81b1bf5, and MemMachine#1736's 1b94f7f, for this PR's
partition registry.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
The query contract said nothing of a match scoring exactly the threshold,
and Qdrant dropped it: the server compares its threshold in single precision
and keeps only scores strictly better than it.

The contract now says a match scoring exactly the threshold is returned.
The Qdrant store sends the server the adjacent single-precision value on
the worse side, and applies the caller's threshold exactly itself; a
threshold beyond single precision is not sent. Both SQLite stores already
keep the match, and each now tests it on every metric it supports.

The same changes as MemMachine#1735's ccba117 and 6bbc131, for this PR's stores.
MemMachine#1736's cc29d27, Milvus's test of the same, is ported with the store
tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…cycle contract's drain

MemMachine#1735's and MemMachine#1736's latest tests, for this PR's partitions:

- Every Qdrant store test runs against a Qdrant server, over REST and gRPC:
  local mode ignores payload indexes and raises its own exceptions where a
  server answers not-found or already-exists, so it is no longer a fixture.
  The tests on a mocked client stay in the default suite.
- Both stores gain tests that partitions and stores of different names keep
  their records apart, that purge rounds reclaim a write landing after a
  round and drain two stores' tombstones alone, that ranking, scores and the
  threshold follow every similarity metric, that a point or entity the
  server refuses on its own raises, that concurrent creations and startups
  agree, that two stores churning one registry keep every partition exact,
  and that seeded operation sequences, and on Milvus random filters, agree
  with a model. The Milvus tests also pin its read consistency and its
  settings with values other than their defaults.
- The lifecycle contract's drain fails after a bounded number of rounds, it
  settles before checking that a new life is empty, and it checks a stale
  upsert by what the partition holds rather than by its registry reads.

The same changes as MemMachine#1735's 30c30f4, e9e71b5, 6c2362b, 4d3241c,
836db21, 252bcd2, 5e8c049, ef8d3ab, 276a034, 4fb5bd4 and
2aabd3d, and MemMachine#1736's 509c235, 7f1eb3b, 6e312e0, 6d1ed1e,
25e7948, 6f36877, d43e9e6, 6dfbf59 and cc29d27, for this PR's
partitions.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
The base store's purge contract says a round that finds the incarnation's
storage missing returns False, but neither store's round checked: once its
native collection was dropped from outside, every round on it raised,
counted against the tombstone, and dead-lettered it after ten.

A Qdrant round that the server answers not found, over REST or gRPC, and a
Milvus round that finds no native collection, now return False: the
collection is gone with everything in it, so the tombstone is retired. Each
store has a test that drops its native collection and drains the purge.

The same handling as MemMachine#1735's and MemMachine#1736's stores, whose rounds already
checked for a missing native collection; MemMachine#1736's 2575af0 tests it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…n, and use serial commas

MemMachine#1734's, MemMachine#1735's, and MemMachine#1736's latest round, for this PR's partition registry
and stores; no behavior changes:

- The queue's `failed_rounds` column is `consecutive_failed_rounds`, in the
  code, the tests, the documents, and the dead-letter log's hint.
- The registry keeps its durations as seconds. `_has_elapsed(seconds,
  since=)` answers whether a duration has passed on the database clock, for
  the retention, the backoff, and the lease, and
  `_purge_retry_backoff_seconds()` computes each tombstone's capped backoff.
- `run_purge_round` runs named steps: `_claim_oldest_due_tombstone` answers a
  `_TombstoneClaim`, an `_UnendedPurgeRound`, or None, and the round's
  outcome goes to `_count_failed_purge_round`,
  `_end_tombstone_claim_after_cancellation`, or `_record_purge_round`.
  `_insert` is `_insert_pending_partition`, `_claim_releases`
  `_tombstone_claim_endings`, and the base store's `_cancellations`
  `_reservation_cancellations`.
- The Milvus filter helpers are named for what they produce, comments are
  shorter, and lists in the documents, comments, and docstrings take a
  serial comma.

The same changes as MemMachine#1734's 2a3d87c and 503687c, MemMachine#1735's 106e106, and
MemMachine#1736's 3681d09 and bc6d4f9, for this PR's partitions.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
A round that never ended was counted by the next claim, which took a
CASE-shaped update that both recorded the lost round and ended its claim;
the count lived apart from the claims that made it.

Each claim now counts its attempt: the claim is a plain update that
increments `consecutive_attempts` (renamed from `consecutive_failed_rounds`),
stamps `claimed_at`, and bumps the generation, and `_TombstoneClaim.attempt`
carries the count. A tombstone is claimable when it has no attempts, when no
claim is open and the backoff has passed since its last failure, or when an
open claim has outlived its lease plus the backoff. A raised round ends its
claim and stamps `last_failed_at`; a cancelled round ends its claim and takes
its attempt back; a round that found records resets the attempts. After
`_MAX_PURGE_ATTEMPTS` attempts a tombstone is dead-lettered, and a last
attempt that raises is reported. A retry logs which attempt it is, and a
raised round's error names its incarnation and attempt. The registry tests,
the purge design document, and the registry design document follow.

The same change as MemMachine#1734's 1732038, for this PR's partition registry.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
`consecutive_attempts` counted the purge rounds claimed since a round last
found records, but its name said only that they were in a row. It is
`attempts_without_progress`, and `_MAX_PURGE_ATTEMPTS` is
`_MAX_PURGE_ATTEMPTS_WITHOUT_PROGRESS`. The column's comment, the claim's
attempt, the backoff parameter's description, the dead-letter log, the
class docstring, the tests, the purge design document, and the registry
design document's table follow; nothing else changes.

The same change as MemMachine#1734's 96a8b5b, for this PR's partition registry.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…urrent claim

The measured cost of the claim came from earlier forms of it, which held a
transaction across the round. The section now gives the figures measured on
2026-10-06 against the claim this registry ports, naming the MemMachine#1734 commits
they were taken at: the backoff scan with 1k, 10k, and 100k tombstones
backing off, and interactive throughput and liveness latency beside two
sweepers on PostgreSQL and on SQLite.

The same change as MemMachine#1734's bab379b, for this PR's purge document.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
MemMachine#1736's review round, for this PR's Milvus store:

- Filter strings reach Milvus as UTF-8, so a character outside the Basic
  Multilingual Plane parses, and an undeclared property's condition requires
  the stored type tag to be one the value compares with.
- Startup's already-exists guard goes, with its mocked-client test: Milvus
  answers a create of an existing collection with the same schema with
  success.
- The store no longer re-sorts search results, which Milvus returns best
  first.
- The mocked-client purge test disposes its registry engine when it fails.

The same changes as MemMachine#1736's 1a7fffa, d6fc219, 9fd7f76, 6fcd8c5, and
c264b3d, for this PR's store. MemMachine#1736's 28e8c50 and b3ef4f1 have no
counterpart here: startup prepares the store's one native collection before
any purge round runs, and the store refuses an unsupported metric at
construction, before anything is reserved.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
The declared path counted an int and a float as comparable either way, so
a float filter value on a declared int property reached Milvus as a float
literal against an INT64 field, which Milvus refuses to parse ("cannot
cast value to Int64", code 1100). A float now compares only with a float
property, and matches no int one, as a value of another type matches
nothing; an int still compares with a float property by value.

The new test filters a declared float property with an int and a declared
int property with a float; it fails with the float counted as comparable
with an int property.

The same change as MemMachine#1736's 31fc7c4, for this PR's store.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…y name

MemMachine#1736's baabf48 creates Milvus native collections at Bounded by name,
and this PR's base carries that into startup's create, so the read level
the purge and the tombstone retention rely on no longer comes from
pymilvus's default. The consistency test now spies create_collection and
checks the level it names, as MemMachine#1736's test does; it fails with the level
dropped from the create. The design document and the store's docstring
say the store creates its one native collection at Bounded.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
MemMachine#1736's f56686f halves a Milvus upsert refused with RESOURCE_EXHAUSTED,
and this PR's base carries the halving into the partition handle's
`_upsert`. Its tests come here in partition terms: the integration test
upserts 1,200 records with a 60,000-character property, about 72 MB, and
fails with RESOURCE_EXHAUSTED without the halving; the mocked-client tests
pin the halving, a single refused entity raising, and a timeout or another
refusal sent once, on a handle `_partition_on` builds, which the mocked
delete test now uses too.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…nt alone

MemMachine#1736's 49b9ec2 drops the offset field beside a declared Milvus
datetime, keeping its UTC instant alone, and this PR's base carries that
into the store. The tests follow in partition terms: the native
collection's fields are exactly the fixed ones and one per declared
property; a declared datetime is stored as its instant in UTC; and a store
whose declared datetime has the longest property key, 32 bytes, writes its
one field, reads it back, and matches the record by its instant, the test
MemMachine#1736's b015229 adds.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
The test of MemMachine#1813, merged into this PR's base, in partition terms: the
server defaults new collections to strict mode, the store's collection
reads back with it off, and a filter on a partition's unindexed property
is served.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A partition stored every property of a record and filtered on any key,
which made a caller's arbitrary keys part of the store's schema: the
SQLite stores kept them in a JSON column and filtered with json_extract,
Milvus the keys its schema did not declare in a JSON field, and a filter
on a key the store never indexed scanned. Since EventMemory routes a
filter on an undeclared key to the segment store, the vector store need
not hold undeclared keys at all.

A partition now stores the properties its store declares and no others.
`upsert` raises UndeclaredPropertyKeyError before anything is sent for a
record naming an undeclared key, and PropertyTypeMismatchError for a
value of another type than its key declares; `query` raises
UndeclaredPropertyKeyError for a filter naming an undeclared key and
UnsupportedFilterError for a node outside the partition's
`supported_filter_nodes`. Both SQLite stores keep one typed, indexed,
nullable column per declared key on the records table (sql_columns.py);
sqlite-vec 0.1.9 rejects NULL in a vec0 metadata column and a declared
key is optional per record, so that store keeps the columns on the
records table and hands the KNN a `rowid IN (SELECT ...)` allowlist,
evaluating the filter during the search instead of after it. Qdrant
drops the JSON copy and keeps a payload field per declared key; Milvus
drops its JSON field and keeps its typed field per declared key.
Datetimes are stored as microseconds since the epoch where a backend has
no datetime type.

Since every key a filter may name is now indexed, the Qdrant store
creates its collection in strict mode (`unindexed_filtering_retrieve`
and `_update` false, Qdrant Cloud's default): a filter on an unindexed key is refused by the server instead
of scanned for. A leaf whose value is of another type than its key
declares matches nothing, as on the SQL stores; the Qdrant compiler
answers it with a filter no point satisfies, since the server would
refuse the condition for the field's index, and Milvus no longer
compares an int with a float key. Local mode does not record the
setting, so a unit test checks the request and integration tests the
server's answer.

`declared_schema_contract.py` states the contract every backend's test
module runs: which records a filtered search admits, over fixtures small
enough that every backend searches them exactly, checked after each
upsert so an approximate index fails on recall, by name, and not on the
filter.

On the registry-backed stores, the checks are the base handle's: `upsert`
runs require_declared_properties in place of the type check it ran, and
`query` runs require_supported_filter against the subclass's
`supported_filter_nodes`, which each subclass now implements. A datetime
column on the SQLite stores has a `tz_<key>` column beside it holding the
UTC offset in seconds, written with the value and read by no filter, so a
stored datetime is the value written, as on Milvus, Qdrant and the segment
store; the microseconds column alone would keep only the instant.

The Qdrant and Milvus design documents describe the declared-only
properties and Qdrant's strict mode.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
A declared datetime on the SQLite stores had a `tz_<key>` column beside its
microseconds column, holding the UTC offset in seconds that no filter
reads. The vector stores keep a datetime property's instant only, as
MemMachine#1736 now does for Milvus and MemMachine#1788 for Qdrant, and the segment store
keeps the offset, so the column, `offset_column_name`, and the value
written to it go: each declared property is one column, and a datetime
stays microseconds since the epoch.

The tests read a stored datetime back as its instant in UTC, and the
roundtrip test checks that the stored instant equals the written one.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
Every embedder MemMachine ships produces vectors meant to be compared by
cosine -- OpenAI hard-coded it, Bedrock defaulted to it, SentenceTransformer
only reported what the model declared -- while every layer that touched a
score paid for the other three metrics in direction flags, threshold
directions, and per-backend tables mapping the enum onto native metric
names. `SimilarityMetric` is gone; scores are cosine similarities in
[-1, 1] and the names say so: `QueryMatch.score` and `SearchMatch.score`
become `cosine_similarity`, and `query(score_threshold=)` becomes
`query(min_cosine_similarity=)`, which no longer needs a direction to be
meaningful and is refused when not finite, as the threshold was. A store
and its partitions no longer have a `similarity_metric`, the schema a
partition is registered under no longer records one, and a search engine
factory takes the dimensions alone. The Bedrock embedder's
`similarity_metric` config key and semantic memory's
`vector_similarity_metric` go with it, and the install and configuration
docs drop them.

The vector graph stores carried a metric per stored embedding, as a
companion property beside every vector; that is gone and `Node.embeddings`
holds plain vectors. NebulaGraph's `cosine()` cannot take `APPROXIMATE` and
its vector indexes offer only L2 and IP. Cosine similarity between unit
vectors is their inner product, so embeddings are normalized on the way in
and compared with `inner_product()` against an IP index.

The cosine half of MemMachine#1598 (`abf92a3a4` on speedkick), re-derived on the
one-collection store: the rest of MemMachine#1598 (queries answer record UUIDs and
scores, no `get`, semantic memory's `vector_uuid`) and all of MemMachine#1603 are in
registry-backed base and its partition handle lose the metric, the stores'
own threshold checks become `require_valid_min_cosine_similarity`, and the
design documents describe cosine scoring and a schema without a metric.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…kend

The filter language is a closed union: Equals, Ordering, In, IsNull,
n-ary And and Or, and Not. Every compiler is an exhaustive match. The
parser maps `!=` and `<>` to Not(Equals), `NOT IN` to Not(In) and
`IS NOT NULL` to Not(IsNull), so a negation means one thing.

A leaf on a field holding no value is false, IsNull on it is true, and
Not is the ordinary complement over the record set. The SQL compilers
render Not as `NOT COALESCE(x, FALSE)`; they compiled `~inner`, which is
three-valued and dropped rows holding no value. Milvus already pushes
Not to the leaves by De Morgan, since its own `not` excludes entities
lacking the field, and Qdrant's must_not already admits them. A
predicate matches only a value of the compared type. The graph stores
and the Neo4j semantic storage get the union only; their NOT over a
comparison on a missing property stays three-valued, since no backend
here can test a change.

Where the compiled output changes: common/filter/sql_filter_util.py and
common/vector_store/sql_columns.py (COALESCE under Not), and the
parser's mapping of `!=`. Where behavior changes: the in-memory test
collection's evaluator matches values by type, the declared-schema
contract gains the complement law and the typed-match test, and two
semantic-storage tests that ordered string values go, since ordering
strings is not expressible in the union. An `In` over no values admits
nothing: every compiler renders it as false, so EventMemory's empty id
lists (session, source, block kind) keep meaning "nothing", and its
complement admits everything. Every other file in this commit changes
node names, import paths and dispatch shape only, with the same output
as before.

The tests the rebuilt base adds use the closed union too, and the segment
store's random context-read test models the new semantics: a condition on
a property with no value is false and a negation is the complement, where
the model had followed SQL's three-valued logic.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
A leaf whose value was of another type than its key declares matched
nothing on every store, each compiler rendering it as a filter no record
satisfies, so a caller who compared a key with the wrong type got an
empty result and no sign of the mistake; and `score > 3` on a float key
matched nothing, since the parser reads `3` as an int. A NaN or an
infinity reached the backends too: it matched every held value on
Qdrant's REST transport, a NaN failed on its gRPC transport, and either
failed on Milvus.

`bind_filter` replaces `require_supported_filter` at every store's query
and returns the filter the store compiles in place of the caller's. It
still refuses an undeclared key and a node outside the store's
`supported_filter_nodes`, and in the filter it returns every value is of
its key's declared type: an int for a float key becomes the float it
equals, and a membership test of ints on a float key the disjunction of
those floats' equalities, since a membership test holds ints or strs.
Any other value of another type than its key declares, a bool for a
number, and a float that is not finite raise PropertyTypeMismatchError,
which `upsert` already raises for a record; the VectorStorePartition
query documents it.

The compilers' branches that rendered a mistyped leaf as no match are
unreachable and go: the Qdrant store's no-match filter, the Milvus
store's filtering of values by type, and the SQL columns compiler's
false branches, keeping only an `In` over no values as false. None of
the three needs the declared schema any more. The Qdrant and Milvus
design documents describe the binding.

The declared-schema contract replaces its no-match test with tests every
store runs: a value of another type and a float that is not finite each
raise from `query`, and an int compares with a float key as the float it
equals, each filter and its complement. The first two fail with the
check in `_require_value` removed, the non-finite cases also with its
finiteness clause removed, and the third fails on Qdrant, whose float
index refuses an int match, with the int-to-float conversion removed
from `_bound_value` or from the membership branch of `_bound_filter`.
The Milvus and Qdrant modules' own no-match tests go, the Qdrant one
with the strict-mode reason it stated.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A record could not hold an int for a float key: upsert refused it as a
type mismatch, though a filter may compare a float key with an int. A
float property that is not finite reached the backends instead, and
each kept it its own way: the SQLite stores held a NaN as no value and
an infinity as written, Qdrant's REST transport held either as no
value, and Qdrant's gRPC transport and Milvus refused the upsert.

`bind_record` replaces `require_declared_properties` at every store's
upsert and returns the record the store writes in place of the
caller's: its keys are declared and every value is of its key's
declared type, an int for a float key becoming the float it equals. Any
other value of another type, a bool for a number, and a float that is
not finite raise PropertyTypeMismatchError, by the same rule the bound
filter follows. The in-memory test partition binds its records too, and
the VectorStorePartition upsert and the error document the rule.

The declared-schema contract checks the rule on every store: an int
written to a float key is stored as the float it equals, read past the
store through a new `stored_value` hook each store's test module
supplies, and matched by a float filter, an int one, and a membership
test of ints; and each mistyped or non-finite value raises from
`upsert`. The per-store tests of refused values lose their
int-for-float case. The first test fails with the int-for-float clause
removed from `_require_value`, and on Qdrant, which keeps an int
payload as an int, with `bind_record` returning the caller's record;
the second fails with the finiteness clause removed, and its bool cases
with a bool counted as an int.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every backend holds an int in 64 signed bits, and each answered a wider
one its own way. A record holding one failed to write on the SQLite
stores, on Qdrant's gRPC transport, and on Milvus, and Qdrant's REST
transport stored it as the largest int64, which an equality filter on
that value then matched. A filter holding one failed on the SQLite
stores and on Milvus, failed an equality on Qdrant, and answered an
ordering on Qdrant with its bound clamped, so `>= 2**63` matched the
largest int64.

The binding both `bind_record` and `bind_filter` apply now refuses an
int outside -2**63 to 2**63 - 1 for any key with
PropertyTypeMismatchError, before anything is sent; the bounds are
MIN_INT_VALUE and MAX_INT_VALUE beside the error. An int for a float key
is held to the same range, so its float always exists.

The declared-schema contract adds out-of-range ints to its mistyped
records and filters, and a test that every store compares the int64
extremes with filters at those extremes. The mistyped cases fail with
the range clause removed from `_require_value`, and the extremes test
with the range made exclusive.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The declared-schema contract gains a test that every store matches
strings holding accented letters, an emoji, quotes, a backslash, and
control characters, by equality and by membership. It fails on Milvus
with `ensure_ascii=False` removed from `_expression_string_literal`, the
UTF-8 literal [vector store scale-out 6/6] ports from MemMachine#1736's 1a7fffa.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Negated vector store filters drop records without the property on the SQLite and Milvus stores

1 participant