Repository navigation
(Depends on #1606) Make a vector store one collection, its handles partitions, and its lookup get_partition on both stores (speedkick) - #1613
Closed
edwinyyyu wants to merge 3 commits into
Conversation
Implements design/event_memory_handoff.md from the tenant-lifecycle branch: the accepted parts of the server redesign that live in EventMemory, the segment store and their data models. Nothing is renamed and nothing outside the event memory is rewired beyond calling the new API. Data models. `Event`, `Segment` and `Derivative` carry `session_id` and `source_id` as first-class nullable fields, copied verbatim down the pipeline. `Block` is an ABC with `kind` as its discriminator and a `render`; the union stays closed over `TextBlock`. `Context` is a mapping from part kind to a registered `ContextPart` (`Author` first; an unregistered kind decodes to `UnknownPart` and round-trips), replacing `ProducerContext`, `NullContext` and the discriminated union. `SearchHit` replaces `ScoredSegmentContext` and `QueryResult`; `Neighborhood` and `EvictionOptions` are added. Reserved keys. Every system value a search filters on at the vector stage sits in the record under a `memmachine_` key (`event_timestamp`, `event_session`, `event_source`, `block_kind`); the collection schema declares the four, and a caller key in the namespace is rejected at `_validate_events`. The derivative's segment and event are not copied into the record: on this base (MemMachine#1598) the vector store answers uuids and scores, and the segment store owns those mappings, so seeds are resolved through `get_segment_uuids_by_derivative_uuids` and eviction reads a stored neighbor's timestamp from its segment. `system_filters.py` owns the translation between the typed parameters (`since`, `until`, `session_ids`, `source_ids`, `block_kinds`) and filter trees on the reserved keys, in both directions, so either answer to "may a caller name a system field in a tree" is a small change at an API boundary. Segment store. `segment_store_sg` gains `session_id`, `source_id` and `block_kind`, projected from the segment at insert, and the ordering index becomes `(incarnation, session_id, timestamp, event_uuid, index, offset)`, the one total order the store exposes. Windows and neighborhoods are confined to the seed's session, with a null session one stream. `get_segment_contexts` takes `before`/`after` and the typed filters; `get_neighbourhoods` returns the neighbors and never the seed, as two lists; `delete_derivatives` unlinks without touching segments. No migration: `startup()` keeps `create_all`, and an existing speedkick database is recreated; schema migration waits for the lifecycle/DDL changes. EventMemory. `encode_events` forgets the batch first, so a repeat leaves one copy, and runs eviction from the agentic_expansion branch, cosine only: batch predecessors, one bounded neighbor query per derivative, a cluster over `target_size` trimmed from its temporal middle, displaced records and their links deleted, skipped ones never written. `query` is the vector stage and returns hits with the seed's index in its window; `rerank` is the second stage, static, for a caller with a reranker; `expand` walks a neighborhood from a segment or event anchor; `render` replaces the string formatters. The reranker and the per-call format options leave the constructor and the call, respectively; the deriver's format is fixed per memory. Server. `LongTermMemory` builds `Event.source_id` from the producer id and adds no context; it reranks after `query` and reads hits. Tests: the branch's neighbor and eviction tests ported to the new shapes on both dialects, plus session confinement, half-open bounds, instant comparison of zoned bounds on SQLite, source and kind filters, and link deletion. Each new store assertion was checked to fail against a mutated store (no session predicate, an inclusive `until`, a filtered neighborhood seed, an unnormalized bound). Rebased 2026-09-10 onto speedkick after MemMachine#1598 merged, adopting its post-review names (`segment_by_derivative`, `seed_cosine_similarities`). `common/property_keys.py` and its test, which MemMachine#1598 did not carry into its merge, are included here. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MuAu353FiSmCJjLX1LWDQW Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
…te undeclared predicates to the segment store (speedkick) The records-and-queries side of the vector store redesign (design/vector_store_handoff.md), written against `VectorStoreCollection` as it is today so the lifecycle work can later replace how a handle is obtained without touching any of this. Declared, not dynamic, indexes. A store declares one `indexed_properties: Mapping[str, PropertyType]` for every collection it holds, from deployment configuration merged with the system keys of the one service that uses it, at the point the store is built for that service (`DatabaseManager.get_vector_store(name, indexed_properties=...)`). `indexed_properties_schema` leaves `VectorStoreCollectionConfig`, and no request creates an index. EventMemory keeps `expected_vector_store_collection_schema` as its declaration of the four reserved keys and raises `InvalidCollectionSchemaError` at construction when the collection's store does not declare them. Services do not share a vector store: a second service whose keys the built store does not declare is refused with `VectorStoreConfigurationError`. Undeclared keys are rejected. `upsert` raises `UndeclaredPropertyKeyError` before anything is sent when a record carries a key the store has not declared, and `PropertyTypeMismatchError` when a declared key holds a value of another type, since a typed column or index cannot hold it without coercing it. `query` raises `UndeclaredPropertyKeyError` when a filter names an undeclared key and `UnsupportedFilterError` when it uses a node outside `supported_filter_nodes`. An undeclared key therefore never exists in the vector store. The filter language. `common/filter/filter_expression.py` is the closed union from the `default` branch (822ccb6): `Equals`, `NotEquals`, `Ordering` over numbers and datetimes, non-empty homogeneous `In`, `IsMissing`, n-ary `And` and `Or`, `Not`. `Comparison` and `IsNull` go; every compiler is an exhaustive `match`. The string parser stays as the server's translation into the union, since the HTTP API still speaks it; a same-operator chain parses to one n-ary node. `split_declared`, `conjoin`, `conjuncts`, `filter_fields` and `filter_nodes` are the tree helpers the routing needs. Ordering strings is no longer expressible, which retires two semantic-storage tests that ordered string values. Semantics. A predicate matches only a record holding a value of the compared type: the SQL column compiler renders a leaf of another type as FALSE rather than letting affinity coerce it, and `Not` is the complement of a match (`NOT COALESCE(x, FALSE)`), so `Not(Equals)` keeps records holding no value while `NotEquals` does not. Qdrant compiles `NotEquals` as "not the value and not empty", since `must_not` alone admits points lacking the field; Milvus pushes `Not` to the leaves by De Morgan, since its own `not` excludes entities lacking the field (measured on Milvus Lite). A datetime is normalized to a UTC instant at node construction. Datetimes. Where a backend has no datetime type, a datetime property is stored as an integer of microseconds since the epoch: the two SQLite stores keep one typed, indexed, nullable column per declared key on the collection's records table (`sql_columns.py`) and store datetimes that way, so a `since` or `until` bound evaluates identically everywhere. sqlite-vec. The handoff planned vec0 metadata columns, on which sqlite-vec evaluates only comparisons joined by AND. sqlite-vec 0.1.9 rejects NULL in a metadata column ("Expected text for TEXT metadata column name, received NULL"), and a declared key is optional per record (`memmachine_event_session` on its own), so the store keeps the declared columns on the records table and hands the KNN `rowid IN (SELECT rowid ... WHERE <filter>)`, which vec0 takes into the search: with k=1 and a filter excluding the nearest rows the admitted row is returned, and every node is evaluated during the search, so the store reports the full node set. Engine-backed store. The declared columns replace the JSON properties column, each with its own index, and the per-candidate SQL predicate compiles over them; MemMachine#1602's selectivity routing is deferred to the lifecycle work and can rebase onto these columns. Both SQLite stores record the schema a collection was created under and raise `IndexedPropertiesMismatchError` when a store built with another schema opens it, since nothing here migrates a column. Clients. `request_timeout` is a required field of `QdrantConf` and `MilvusConf`, passed to the client at construction. Qdrant's `hnsw_config`, `optimizers_config` and `quantization_config` come over from the `default` branch (a0753d3) as plain mappings validated against the qdrant models when the store is built; `hnsw_config.m` must be 0 or unset, and `payload_m` is the knob. Milvus drops the properties JSON field nothing read. EventMemory. One plan in `query`: the vector store gets the system predicates and the part of the caller's filter naming declared keys; the segment store, which holds every property, applies the whole filter to the seeds and their windows afterward. When the filter has an undeclared part, a seed the segment store drops leaves the search short and the vector limit is widened by four each time, up to `limit * FilterOptions.max_overfetch_factor`, where the search returns what survived; with no undeclared part the first query is the last. An empty id or kind list admits nothing and issues no query. Every count is a maximum. Not in this change: keys, registries, strict create, logical delete and purge, stateless handles, one container per embedder, the SQLite stores on shared tables, and the removal of content-addressed names and Qdrant shard keys (the collections-and-keys side of the seam). Native Qdrant and Milvus collection names still hash the collection config, which no longer carries a schema, so existing native collections are not reused. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01DtbsPQafKpBUn7ksnDU2UY
…ookup get_partition on both stores (speedkick)
One store is one collection. A `VectorStore` is built for one collection,
named at construction with its vector dimensions and its declared
`indexed_properties`, and the composition root builds one store per cell
of the purpose-by-embedder matrix (`long_term_memory__<embedder>`,
`semantic_memory__<embedder>`). The collection name is what keeps two
stores over one engine or one client apart: it names the native Qdrant
and Milvus collections, the SQLite table prefixes, and each store's
registry. `namespace`, `VectorStoreCollectionConfig`, the content-addressed
native names and the config-mismatch error go with it.
Handles are partitions. `VectorStorePartition` is bound to its key and owns
nothing, so a caller builds one, drops it and builds another at will; the
ABC is the segment store's shape:
provision() # idempotent DDL
create_partition(partition_key) # strict
get_partition(partition_key) -> Partition | None # None when absent
delete_partition(partition_key) # idempotent
`get_partition` reads the registry row and binds; a lookup that does I/O
is fine as long as nothing has to be closed, and staleness stays a
property of a handle already held, raised by its operations. The segment
store gets the same verb: `open_partition` becomes `get_partition`, and
`open_or_create_partition` and `close_partition` go from both stores.
Create-if-absent is the composition root's, as `get_or_create_partition`
beside the vector store ABC and its counterpart in the service locator;
the segment store's config-mismatch error had no raiser left.
Keys stay `str` under today's identifier contract. The decided
`partition(key: UUID)` needs keys minted once and never reused, which the
tenant service's tombstone provides; until then a UUID derived from the
session id would be reused across delete and recreate. Logical delete,
purge and the per-operation registry read are likewise the lifecycle
work's; `delete_partition` deletes synchronously as `delete_collection`
did, and the composition root calls `provision()` when it builds a store
until the schema command owns it.
Each partition records the dimensions and declared schema it was created
under, in the SQLite registry rows and the Qdrant and Milvus registry
points, and `get_partition` raises `VectorStorePartitionSchemaMismatchError`
when a store built with others opens it. The SQLite registries are shared
tables keyed by collection and partition, so stores of different
collections share one engine; records tables and index files stay per
partition, named by collection and key.
`DatabaseManager.get_vector_store(backend, collection=, vector_dimensions=,
indexed_properties=)` builds and caches one store per (backend,
collection), provisions and starts it, and refuses the same collection
asked for with other dimensions or keys; SQLite backends share one engine
across their collections. The "second service refused" guard is gone,
since a cell is a store by construction.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01DtbsPQafKpBUn7ksnDU2UY
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 14, 2026
…, and close the filter union (speedkick)
A vector store is built for one collection, named at construction with its
vector dimensions and its declared `indexed_properties`; the composition
root builds one store per purpose-by-embedder cell. Within it, a partition
holds one tenant's records, and `VectorStorePartition` is the handle a
data consumer holds. Lifecycle on both storage components:
provision() # idempotent DDL
create_partition(key) # strict
get_partition(key) -> Partition | None # None when absent
delete_partition(key) # idempotent
Declared keys are typed, indexed columns of the SQLite records tables and
payload indexes on Qdrant and Milvus; a record or a filter naming an
undeclared key is rejected before anything is sent, so an undeclared key
never exists in a vector store. EventMemory routes the declared part of a
filter to the vector store and the rest to the segment store's
post-filter, widening the search up to `max_overfetch_factor`.
The filter language is a closed union: Equals, Ordering, In, IsNull, And,
Or, Not. `!=` parses as Not(Equals); Not is the complement on every
backend, so a record holding no value passes a negated predicate.
Datetimes are stored as microseconds since the epoch where a backend has
no datetime type. Each backend states its `supported_filter_nodes`.
Folds MemMachine#1613 and MemMachine#1615 into MemMachine#1606 and moves the whole off MemMachine#1597: the
routing is written against speedkick's EventMemory, and the session,
source and expansion parameters return with MemMachine#1597.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
Contributor
Author
|
Folded into #1606, which now sits directly on speedkick. The old tip is kept as tag pre-reorg-1613 locally. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose of the change
Lands the store-as-collection decisions on both storage components without pulling in the tenant lifecycle. Stacked on #1606 (its commit shows in this diff until it merges); stacked on #1597 through it.
One store is one collection. A
VectorStoreis built for one collection, named at construction with its vector dimensions and its declaredindexed_properties. The composition root builds one store per cell of the purpose-by-embedder matrix (long_term_memory__<embedder>,semantic_memory__<embedder>), and the collection name is what keeps two stores over one engine or client apart: it names the native Qdrant and Milvus collections, the SQLite table prefixes, and each store's registry.namespace,VectorStoreCollectionConfig, the content-addressed native names and the config-mismatch error go.Handles are partitions, and the verb is
get_partitionon both stores.get_partitionreads the registry row and binds; a lookup that does I/O is fine as long as nothing has to be closed, and staleness stays a property of a handle already held, raised by its operations. The segment store'sopen_partitionbecomesget_partition;open_or_create_partitionandclose_partitiongo from both stores, and create-if-absent is the composition root's (get_or_create_partitionbeside the vector store ABC, and its counterpart in the service locator).Each partition records what it was created under (dimensions and declared schema, in the SQLite registry rows and the Qdrant and Milvus registry points), and
get_partitionraisesVectorStorePartitionSchemaMismatchErrorwhen a store built with others opens it. The SQLite registries are shared tables keyed by collection and partition, so stores of different collections share one engine.DatabaseManager.get_vector_store(backend, collection=, vector_dimensions=, indexed_properties=)builds and caches one store per (backend, collection), provisions and starts it, and refuses the same collection asked for with other dimensions or keys. The "second service refused" guard from [vector store 1/12] Remove per-project filterable properties (speedkick) #1606 is gone, since a cell is a store by construction.What stays with the lifecycle work
strunder today's identifier contract. The decidedpartition(key: UUID)needs keys minted once and never reused, which the tenant service's tombstone provides; a UUID derived from the session id would be reused across delete and recreate.delete_partitiondeletes synchronously asdelete_collectiondid.provision()when it builds a store until then.Tests
The four store modules, the segment store module, the resource manager and wiring tests are rewritten to the shape: fixtures build a store per collection and provision it;
create_collection(namespace, name, config)calls becomecreate_partition(name), and so on. Tests of removed paths go: open-or-create idempotency and config mismatch on both stores, the segment store's lost-race retry arm (that loop lived in the removed method; the composition root's create-if-absent makes one attempt), and the namespace tests. Added: two stores of different collections over one engine are isolated; provisioning creates the payload indexes on a Qdrant collection that already exists and survives two concurrent provisioners (integration); the same collection asked for with other dimensions or keys is refused; collections on one backend share its engine.uv run pytest packages/server/server_tests: 2016 passed, 2 skipped.ruff checkandruff format --checkclean.ty check packagesreports only the diagnostics already on the base.🤖 Generated with Claude Code
https://claude.ai/code/session_01DtbsPQafKpBUn7ksnDU2UY