Skip to content

(Depends on #1606) Make a vector store one collection, its handles partitions, and its lookup get_partition on both stores (speedkick) - #1613

Closed
edwinyyyu wants to merge 3 commits into
MemMachine:speedkickfrom
edwinyyyu:feat/vector-store-partitions-speedkick
Closed

edwinyyyu wants to merge 3 commits into
MemMachine:speedkickfrom
edwinyyyu:feat/vector-store-partitions-speedkick

Conversation

@edwinyyyu

Copy link
Copy Markdown
Contributor

Purpose of the change

Lands the store-as-collection decisions on both storage components without pulling in the tenant lifecycle. Stacked on #1606 (its commit shows in this diff until it merges); stacked on #1597 through it.

  • One store is one collection. A VectorStore is built for one collection, named at construction with its vector dimensions and its declared indexed_properties. The composition root builds one store per cell of the purpose-by-embedder matrix (long_term_memory__<embedder>, semantic_memory__<embedder>), and the collection name is what keeps two stores over one engine or client apart: it names the native Qdrant and Milvus collections, the SQLite table prefixes, and each store's registry. namespace, VectorStoreCollectionConfig, the content-addressed native names and the config-mismatch error go.

  • Handles are partitions, and the verb is get_partition on both stores.

    provision()                                        # idempotent DDL
    create_partition(partition_key)                    # strict
    get_partition(partition_key) -> Partition | None   # None when absent
    delete_partition(partition_key)                    # idempotent
    

    get_partition reads the registry row and binds; a lookup that does I/O is fine as long as nothing has to be closed, and staleness stays a property of a handle already held, raised by its operations. The segment store's open_partition becomes get_partition; open_or_create_partition and close_partition go from both stores, and create-if-absent is the composition root's (get_or_create_partition beside the vector store ABC, and its counterpart in the service locator).

  • Each partition records what it was created under (dimensions and declared schema, in the SQLite registry rows and the Qdrant and Milvus registry points), and get_partition raises VectorStorePartitionSchemaMismatchError when a store built with others opens it. The SQLite registries are shared tables keyed by collection and partition, so stores of different collections share one engine.

  • DatabaseManager.get_vector_store(backend, collection=, vector_dimensions=, indexed_properties=) builds and caches one store per (backend, collection), provisions and starts it, and refuses the same collection asked for with other dimensions or keys. The "second service refused" guard from [vector store 1/12] Remove per-project filterable properties (speedkick) #1606 is gone, since a cell is a store by construction.

What stays with the lifecycle work

  • Keys stay str under today's identifier contract. The decided partition(key: UUID) needs keys minted once and never reused, which the tenant service's tombstone provides; a UUID derived from the session id would be reused across delete and recreate.
  • Logical delete, purge and the per-operation registry read. delete_partition deletes synchronously as delete_collection did.
  • The schema command. The composition root calls provision() when it builds a store until then.
  • Shared tables for the SQLite stores' records; records tables and index files stay per partition, named by collection and key.

Tests

The four store modules, the segment store module, the resource manager and wiring tests are rewritten to the shape: fixtures build a store per collection and provision it; create_collection(namespace, name, config) calls become create_partition(name), and so on. Tests of removed paths go: open-or-create idempotency and config mismatch on both stores, the segment store's lost-race retry arm (that loop lived in the removed method; the composition root's create-if-absent makes one attempt), and the namespace tests. Added: two stores of different collections over one engine are isolated; provisioning creates the payload indexes on a Qdrant collection that already exists and survives two concurrent provisioners (integration); the same collection asked for with other dimensions or keys is refused; collections on one backend share its engine.

uv run pytest packages/server/server_tests: 2016 passed, 2 skipped. ruff check and ruff format --check clean. ty check packages reports only the diagnostics already on the base.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DtbsPQafKpBUn7ksnDU2UY

edwinyyyu and others added 3 commits September 10, 2026 12:19
Implements design/event_memory_handoff.md from the tenant-lifecycle
branch: the accepted parts of the server redesign that live in
EventMemory, the segment store and their data models. Nothing is renamed
and nothing outside the event memory is rewired beyond calling the new
API.

Data models. `Event`, `Segment` and `Derivative` carry `session_id` and
`source_id` as first-class nullable fields, copied verbatim down the
pipeline. `Block` is an ABC with `kind` as its discriminator and a
`render`; the union stays closed over `TextBlock`. `Context` is a mapping
from part kind to a registered `ContextPart` (`Author` first; an
unregistered kind decodes to `UnknownPart` and round-trips), replacing
`ProducerContext`, `NullContext` and the discriminated union. `SearchHit`
replaces `ScoredSegmentContext` and `QueryResult`; `Neighborhood` and
`EvictionOptions` are added.

Reserved keys. Every system value a search filters on at the vector
stage sits in the record under a `memmachine_` key (`event_timestamp`,
`event_session`, `event_source`, `block_kind`); the collection schema
declares the four, and a caller key in the namespace is rejected at
`_validate_events`. The derivative's segment and event are not copied
into the record: on this base (MemMachine#1598) the vector store answers uuids and
scores, and the segment store owns those mappings, so seeds are resolved
through `get_segment_uuids_by_derivative_uuids` and eviction reads a
stored neighbor's timestamp from its segment. `system_filters.py` owns
the translation between the typed parameters (`since`, `until`,
`session_ids`, `source_ids`, `block_kinds`) and filter trees on the
reserved keys, in both directions, so either answer to "may a caller
name a system field in a tree" is a small change at an API boundary.

Segment store. `segment_store_sg` gains `session_id`, `source_id` and
`block_kind`, projected from the segment at insert, and the ordering
index becomes `(incarnation, session_id, timestamp, event_uuid, index,
offset)`, the one total order the store exposes. Windows and
neighborhoods are confined to the seed's session, with a null session
one stream. `get_segment_contexts` takes `before`/`after` and the typed
filters; `get_neighbourhoods` returns the neighbors and never the seed,
as two lists; `delete_derivatives` unlinks without touching segments.
No migration: `startup()` keeps `create_all`, and an existing speedkick
database is recreated; schema migration waits for the lifecycle/DDL
changes.

EventMemory. `encode_events` forgets the batch first, so a repeat leaves
one copy, and runs eviction from the agentic_expansion branch, cosine
only: batch predecessors, one bounded neighbor query per derivative, a
cluster over `target_size` trimmed from its temporal middle, displaced
records and their links deleted, skipped ones never written. `query` is
the vector stage and returns hits with the seed's index in its window;
`rerank` is the second stage, static, for a caller with a reranker;
`expand` walks a neighborhood from a segment or event anchor; `render`
replaces the string formatters. The reranker and the per-call format
options leave the constructor and the call, respectively; the deriver's
format is fixed per memory.

Server. `LongTermMemory` builds `Event.source_id` from the producer id
and adds no context; it reranks after `query` and reads hits.

Tests: the branch's neighbor and eviction tests ported to the new shapes
on both dialects, plus session confinement, half-open bounds, instant
comparison of zoned bounds on SQLite, source and kind filters, and link
deletion. Each new store assertion was checked to fail against a mutated store (no session
predicate, an inclusive `until`, a filtered neighborhood seed, an
unnormalized bound).

Rebased 2026-09-10 onto speedkick after MemMachine#1598 merged, adopting its
post-review names (`segment_by_derivative`, `seed_cosine_similarities`).
`common/property_keys.py` and its test, which MemMachine#1598 did not carry into
its merge, are included here.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MuAu353FiSmCJjLX1LWDQW
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
…te undeclared predicates to the segment store (speedkick)

The records-and-queries side of the vector store redesign
(design/vector_store_handoff.md), written against `VectorStoreCollection`
as it is today so the lifecycle work can later replace how a handle is
obtained without touching any of this.

Declared, not dynamic, indexes. A store declares one
`indexed_properties: Mapping[str, PropertyType]` for every collection it
holds, from deployment configuration merged with the system keys of the
one service that uses it, at the point the store is built for that
service (`DatabaseManager.get_vector_store(name, indexed_properties=...)`).
`indexed_properties_schema` leaves `VectorStoreCollectionConfig`, and no
request creates an index. EventMemory keeps
`expected_vector_store_collection_schema` as its declaration of the four
reserved keys and raises `InvalidCollectionSchemaError` at construction
when the collection's store does not declare them. Services do not share
a vector store: a second service whose keys the built store does not
declare is refused with `VectorStoreConfigurationError`.

Undeclared keys are rejected. `upsert` raises `UndeclaredPropertyKeyError`
before anything is sent when a record carries a key the store has not
declared, and `PropertyTypeMismatchError` when a declared key holds a
value of another type, since a typed column or index cannot hold it
without coercing it. `query` raises `UndeclaredPropertyKeyError` when a
filter names an undeclared key and `UnsupportedFilterError` when it uses a
node outside `supported_filter_nodes`. An undeclared key therefore never
exists in the vector store.

The filter language. `common/filter/filter_expression.py` is the closed
union from the `default` branch (822ccb6): `Equals`, `NotEquals`,
`Ordering` over numbers and datetimes, non-empty homogeneous `In`,
`IsMissing`, n-ary `And` and `Or`, `Not`. `Comparison` and `IsNull` go;
every compiler is an exhaustive `match`. The string parser stays as the
server's translation into the union, since the HTTP API still speaks it;
a same-operator chain parses to one n-ary node. `split_declared`,
`conjoin`, `conjuncts`, `filter_fields` and `filter_nodes` are the tree
helpers the routing needs. Ordering strings is no longer expressible,
which retires two semantic-storage tests that ordered string values.

Semantics. A predicate matches only a record holding a value of the
compared type: the SQL column compiler renders a leaf of another type as
FALSE rather than letting affinity coerce it, and `Not` is the complement
of a match (`NOT COALESCE(x, FALSE)`), so `Not(Equals)` keeps records
holding no value while `NotEquals` does not. Qdrant compiles `NotEquals`
as "not the value and not empty", since `must_not` alone admits points
lacking the field; Milvus pushes `Not` to the leaves by De Morgan, since
its own `not` excludes entities lacking the field (measured on Milvus
Lite). A datetime is normalized to a UTC instant at node construction.

Datetimes. Where a backend has no datetime type, a datetime property is
stored as an integer of microseconds since the epoch: the two SQLite
stores keep one typed, indexed, nullable column per declared key on the
collection's records table (`sql_columns.py`) and store datetimes that
way, so a `since` or `until` bound evaluates identically everywhere.

sqlite-vec. The handoff planned vec0 metadata columns, on which sqlite-vec
evaluates only comparisons joined by AND. sqlite-vec 0.1.9 rejects NULL in
a metadata column ("Expected text for TEXT metadata column name, received
NULL"), and a declared key is optional per record (`memmachine_event_session`
on its own), so the store keeps the declared columns on the records table
and hands the KNN `rowid IN (SELECT rowid ... WHERE <filter>)`, which vec0
takes into the search: with k=1 and a filter excluding the nearest rows the
admitted row is returned, and every node is evaluated during the search,
so the store reports the full node set.

Engine-backed store. The declared columns replace the JSON properties
column, each with its own index, and the per-candidate SQL predicate
compiles over them; MemMachine#1602's selectivity routing is deferred to the
lifecycle work and can rebase onto these columns. Both SQLite stores
record the schema a collection was created under and raise
`IndexedPropertiesMismatchError` when a store built with another schema
opens it, since nothing here migrates a column.

Clients. `request_timeout` is a required field of `QdrantConf` and
`MilvusConf`, passed to the client at construction. Qdrant's
`hnsw_config`, `optimizers_config` and `quantization_config` come over
from the `default` branch (a0753d3) as plain mappings validated against
the qdrant models when the store is built; `hnsw_config.m` must be 0 or
unset, and `payload_m` is the knob. Milvus drops the properties JSON
field nothing read.

EventMemory. One plan in `query`: the vector store gets the system
predicates and the part of the caller's filter naming declared keys; the
segment store, which holds every property, applies the whole filter to
the seeds and their windows afterward. When the filter has an undeclared
part, a seed the segment store drops leaves the search short and the
vector limit is widened by four each time, up to
`limit * FilterOptions.max_overfetch_factor`, where the search returns
what survived; with no undeclared part the first query is the last. An
empty id or kind list admits nothing and issues no query. Every count is
a maximum.

Not in this change: keys, registries, strict create, logical delete and
purge, stateless handles, one container per embedder, the SQLite stores
on shared tables, and the removal of content-addressed names and Qdrant
shard keys (the collections-and-keys side of the seam). Native Qdrant and
Milvus collection names still hash the collection config, which no longer
carries a schema, so existing native collections are not reused.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01DtbsPQafKpBUn7ksnDU2UY
…ookup get_partition on both stores (speedkick)

One store is one collection. A `VectorStore` is built for one collection,
named at construction with its vector dimensions and its declared
`indexed_properties`, and the composition root builds one store per cell
of the purpose-by-embedder matrix (`long_term_memory__<embedder>`,
`semantic_memory__<embedder>`). The collection name is what keeps two
stores over one engine or one client apart: it names the native Qdrant
and Milvus collections, the SQLite table prefixes, and each store's
registry. `namespace`, `VectorStoreCollectionConfig`, the content-addressed
native names and the config-mismatch error go with it.

Handles are partitions. `VectorStorePartition` is bound to its key and owns
nothing, so a caller builds one, drops it and builds another at will; the
ABC is the segment store's shape:

    provision()                                        # idempotent DDL
    create_partition(partition_key)                    # strict
    get_partition(partition_key) -> Partition | None   # None when absent
    delete_partition(partition_key)                    # idempotent

`get_partition` reads the registry row and binds; a lookup that does I/O
is fine as long as nothing has to be closed, and staleness stays a
property of a handle already held, raised by its operations. The segment
store gets the same verb: `open_partition` becomes `get_partition`, and
`open_or_create_partition` and `close_partition` go from both stores.
Create-if-absent is the composition root's, as `get_or_create_partition`
beside the vector store ABC and its counterpart in the service locator;
the segment store's config-mismatch error had no raiser left.

Keys stay `str` under today's identifier contract. The decided
`partition(key: UUID)` needs keys minted once and never reused, which the
tenant service's tombstone provides; until then a UUID derived from the
session id would be reused across delete and recreate. Logical delete,
purge and the per-operation registry read are likewise the lifecycle
work's; `delete_partition` deletes synchronously as `delete_collection`
did, and the composition root calls `provision()` when it builds a store
until the schema command owns it.

Each partition records the dimensions and declared schema it was created
under, in the SQLite registry rows and the Qdrant and Milvus registry
points, and `get_partition` raises `VectorStorePartitionSchemaMismatchError`
when a store built with others opens it. The SQLite registries are shared
tables keyed by collection and partition, so stores of different
collections share one engine; records tables and index files stay per
partition, named by collection and key.

`DatabaseManager.get_vector_store(backend, collection=, vector_dimensions=,
indexed_properties=)` builds and caches one store per (backend,
collection), provisions and starts it, and refuses the same collection
asked for with other dimensions or keys; SQLite backends share one engine
across their collections. The "second service refused" guard is gone,
since a cell is a store by construction.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01DtbsPQafKpBUn7ksnDU2UY
@edwinyyyu edwinyyyu changed the title Make a vector store one collection, its handles partitions, and its lookup get_partition on both stores (speedkick) (Depends on #1606) Make a vector store one collection, its handles partitions, and its lookup get_partition on both stores (speedkick) Sep 11, 2026
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 14, 2026
…, and close the filter union (speedkick)

A vector store is built for one collection, named at construction with its
vector dimensions and its declared `indexed_properties`; the composition
root builds one store per purpose-by-embedder cell. Within it, a partition
holds one tenant's records, and `VectorStorePartition` is the handle a
data consumer holds. Lifecycle on both storage components:

    provision()                                        # idempotent DDL
    create_partition(key)                              # strict
    get_partition(key) -> Partition | None             # None when absent
    delete_partition(key)                              # idempotent

Declared keys are typed, indexed columns of the SQLite records tables and
payload indexes on Qdrant and Milvus; a record or a filter naming an
undeclared key is rejected before anything is sent, so an undeclared key
never exists in a vector store. EventMemory routes the declared part of a
filter to the vector store and the rest to the segment store's
post-filter, widening the search up to `max_overfetch_factor`.

The filter language is a closed union: Equals, Ordering, In, IsNull, And,
Or, Not. `!=` parses as Not(Equals); Not is the complement on every
backend, so a record holding no value passes a negated predicate.
Datetimes are stored as microseconds since the epoch where a backend has
no datetime type. Each backend states its `supported_filter_nodes`.

Folds MemMachine#1613 and MemMachine#1615 into MemMachine#1606 and moves the whole off MemMachine#1597: the
routing is written against speedkick's EventMemory, and the session,
source and expansion parameters return with MemMachine#1597.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu

Copy link
Copy Markdown
Contributor Author

Folded into #1606, which now sits directly on speedkick. The old tip is kept as tag pre-reorg-1613 locally.

@edwinyyyu edwinyyyu closed this Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant