Repository navigation
Overhaul segment store: shared tables with incarnation-scoped tenant keys (port of #1548) - #1661
Merged
edwinyyyu merged 14 commits intoSep 25, 2026
Conversation
This was referenced Sep 16, 2026
Closed
Merged
[session storage 2/2] Remove open-or-create from both stores, and close from the segment store
#1625
Draft
Closed
Draft
edwinyyyu
force-pushed
the
port/segment-store-shared-tables-main
branch
from
September 17, 2026 17:39
ddd65aa to
9196973
Compare
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
The existing context tests pass with the lateral's per-seed walk left uncorrelated to its seed (only one filtered test, and it has one seed) and with segments sharing a timestamp ordered by the wrong tie-breakers (every test gives each segment its own timestamp). The new test reads random contexts, several seeds at a time, with filters of every node type, from a partition where many segments share a timestamp and another partition holds segments at the same times, and compares each result with an in-memory model that evaluates filters in SQL's logic. Every partition is far smaller than the filtered read's window, so the model has no bound. It passes on both dialects here and at MemMachine#1661's tip, and fails on both with the window ordered the same way on either side, with the filter resolved against the table instead of the window, and with the tie-breakers reordered, and on PostgreSQL with the walk uncorrelated. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
Building the window's statement through an ORM alias adapted every column the filter and the ordering read from it, and the walk went through the mapped class. With a property filter that made building a PostgreSQL request's two statements, with SQLAlchemy's cache key, cost 1.6-1.8 ms of Python against 0.86 ms at MemMachine#1661's tip. The walk and the window now use the table's and the window's own columns, and SQLite's per-seed statements load segment rows through select(SegmentRow).from_statement(), since building them from plain rows cost more than ORM loading. The SQL is byte-identical on both dialects. A filtered request now takes 1.15-1.21 ms to build, of which about 0.3 ms is the window itself: its subquery's columns and the larger statement to hash. Filtered reads on PostgreSQL partitions of 200 and 2,000 segments, 20 seeds, went from 4-28% slower than MemMachine#1661's tip to 1-9%; SQLite reads are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
sqlalchemy.sql.elements is an internal module; the same class is exported from sqlalchemy and documented as sqlalchemy.sql.expression.ColumnElement. Same object, import path only. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
…emMachine#1712) A context read with a property filter selected, per seed and direction, the first matching rows in walk order: ORDER BY the walk with the filter in the WHERE clause and LIMIT the context size. The planner has no statistics for property predicates; it estimates about one match per seed, costs the ordered walk as reading the seed's whole side, and on PostgreSQL plans a bitmap over that side and a sort instead. The read then grew with the partition rather than with the context asked for. With a filter, the context now comes from the seed's next 1,024 segments in walk order, read with no filter, and the filter and the context size apply over those rows. The inner read has no filter to misestimate, so it stays an ordered index scan, and it stops once the context is found or the window ends. A match further than 1,024 segments from the seed is no longer context; the segment store ABC now allows that bound. Unfiltered reads keep their statement. SQLite's per-seed reads take the same window, so both dialects return the same context. Measured against MemMachine#1661's tip, 20 seeds per request with 2 segments back and 4 forward, the rows of 21 partitions interleaved: - PostgreSQL 16: a filtered request takes 8-12 ms instead of 4.5-7.1 s on a 1,000,000-segment partition, and 8-13 ms instead of 103-115 ms on 20,000-segment ones; unfiltered, 7.6 ms against 7.8 ms. - SQLite, where the plan never read the whole side: filters matching 20% or 1% take 1.1-1.2x as long as with the previous commit alone (14.1 vs 12.4 ms, 28.4 vs 24.2 ms), while a property that only the seeds carry takes 36 ms instead of 602 ms on a 200,000-segment partition, since the walk no longer runs toward the partition's end. - For the same seeds, both dialects returned the same context as before, except where the bound applies: 12 forward with a filter matching every 100th segment returns the 10 matches within 1,024 segments. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
The existing context tests pass with the lateral's per-seed walk left uncorrelated to its seed (only one filtered test, and it has one seed) and with segments sharing a timestamp ordered by the wrong tie-breakers (every test gives each segment its own timestamp). The new test reads random contexts, several seeds at a time, with filters of every node type, from a partition where many segments share a timestamp and another partition holds segments at the same times, and compares each result with an in-memory model that evaluates filters in SQL's logic. Every partition is far smaller than the filtered read's window, so the model has no bound. It passes on both dialects here and at MemMachine#1661's tip, and fails on both with the window ordered the same way on either side, with the filter resolved against the table instead of the window, and with the tie-breakers reordered, and on PostgreSQL with the walk uncorrelated. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
Building the window's statement through an ORM alias adapted every column the filter and the ordering read from it, and the walk went through the mapped class. With a property filter that made building a PostgreSQL request's two statements, with SQLAlchemy's cache key, cost 1.6-1.8 ms of Python against 0.86 ms at MemMachine#1661's tip. The walk and the window now use the table's and the window's own columns, and SQLite's per-seed statements load segment rows through select(SegmentRow).from_statement(), since building them from plain rows cost more than ORM loading. The SQL is byte-identical on both dialects. A filtered request now takes 1.15-1.21 ms to build, of which about 0.3 ms is the window itself: its subquery's columns and the larger statement to hash. Filtered reads on PostgreSQL partitions of 200 and 2,000 segments, 20 seeds, went from 4-28% slower than MemMachine#1661's tip to 1-9%; SQLite reads are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
validate_identifier matched `^[a-z0-9_]+$`, and `$` also matches before a trailing newline, so "name\n" passed as a namespace, a collection name, and now a vector store name that becomes part of the registry's table names. It uses fullmatch, as MemMachine#1661 does for the segment store's partition keys. The event backend's service locator already hashes such a session id: it checks session ids with the segment store's validator, which uses fullmatch since MemMachine#1661. Tests, each verified to fail with `match` restored: a vector store name with a trailing newline is refused, and a namespace or collection name with one is refused by both stores. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
The registry is built and tested on PostgreSQL and SQLite only. On another dialect it misbehaves rather than failing at construction: the retention cutoff is PostgreSQL's interval arithmetic outside SQLite, MySQL has no DELETE ... RETURNING, and on SQL Server SQLAlchemy drops the claims' FOR UPDATE SKIP LOCKED without a word. The constructor now refuses any other dialect with a ValueError, beside its vector store name check, as MemMachine#1661 does in the segment store's engine validator. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
…registry for Equivalents of the segment store tests MemMachine#1661 adds, where the registry has the same behavior, each verified to fail with that behavior removed and to pass with it: - a claim skips a tombstone another purger holds (PostgreSQL; fails with SKIP LOCKED dropped), as test_purge_skips_entries_claimed_by_concurrent_purger; - racing deletions of one collection queue one tombstone and leave the name free, and a deletion racing another process's waits and then queues nothing (fail with a read before the DELETE), as test_concurrent_partition_deletes_are_clean and test_concurrent_remote_delete_yields_single_queue_entry; - a mint colliding with a live incarnation is minted again, not reported as a taken name (fails with every integrity error read as a taken name), as test_incarnation_colliding_with_live_partition_is_never_reused; - a mint colliding with a deletion in flight checks the queue after its insert (fails with the check moved before it), as test_mint_detects_collision_with_concurrent_deletion and its SQLite counterpart, staged by the insert being issued rather than by a sleep; - a round claims one tombstone (fails without the LIMIT), as test_purge_claims_queue_entries_incrementally; - a persistent database error surfaces with its cause chained (fails with the cause dropped), as test_persistent_integrity_error_surfaces_with_cause. The existing test that an incarnation awaiting purge is never minted again, the equivalent of test_incarnation_with_garbage_left_is_never_reused, was verified the same way. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
…cess Milvus Lite does not arbitrate two creations of one collection racing in one process: the loser fails with "[Errno 17] File exists", which is not an already-exists error, so the logical creation raised. Every logical collection of a namespace and configuration shares its native collection, so two sessions created at once before it exists race on it (on main too, for different names; the per-name locks this PR removed had covered the same name). The store now takes a per-process lock per native collection around the existence check and the creation; the later creator finds the collection. Creation stays idempotent across processes as before. The contract gains a lifecycle churn test, the equivalent of MemMachine#1661's test_lifecycle_churn_completes_without_database_errors: concurrent create, open-or-create, open and delete of a few names raise nothing but the domain's own outcomes. It found this on Milvus Lite and fails there without the lock. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
…ifecycle for Equivalents of the segment store tests MemMachine#1661 adds, in the lifecycle contract both stores run (Qdrant in-memory, REST and gRPC; Milvus Lite), each verified to fail with the behavior removed and to pass with it: - a read checks the registry once, before it, and a write twice, around it (fails with a check after a read, or without the check after a write), as test_reads_check_liveness_inside_the_data_statement pins the segment store's registry round trips per operation; - a creation that fails at the native collection registers nothing, for create_collection and open_or_create_collection (fails with the registry written first), as test_unloadable_codec_config_commits_no_registry_row; - open-or-create gives up after a bounded number of lost races (fails without the bound), and creates again when the winner it lost to is gone (fails if that case raises), as the lost-race arm of test_persistent_mint_failure_raises_instead_of_looping; - deletion leaves the records for the purge (fails if deletion reclaims them itself), as test_delete_partition_touches_only_registry_and_queue: the purge test asserted the records unreachable but not still held. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
validate_identifier matched `^[a-z0-9_]+$`, and `$` also matches before a trailing newline, so "name\n" passed as a namespace, a collection name, and now a vector store name that becomes part of the registry's table names. It uses fullmatch, as MemMachine#1661 does for the segment store's partition keys. The event backend's service locator already hashes such a session id: it checks session ids with the segment store's validator, which uses fullmatch since MemMachine#1661. Tests, each verified to fail with `match` restored: a vector store name with a trailing newline is refused, and a namespace or collection name with one is refused by both stores. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
The registry is built and tested on PostgreSQL and SQLite only. On another dialect it misbehaves rather than failing at construction: the retention cutoff is PostgreSQL's interval arithmetic outside SQLite, MySQL has no DELETE ... RETURNING, and on SQL Server SQLAlchemy drops the claims' FOR UPDATE SKIP LOCKED without a word. The constructor now refuses any other dialect with a ValueError, beside its vector store name check, as MemMachine#1661 does in the segment store's engine validator. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
…registry for Equivalents of the segment store tests MemMachine#1661 adds, where the registry has the same behavior, each verified to fail with that behavior removed and to pass with it: - a claim skips a tombstone another purger holds (PostgreSQL; fails with SKIP LOCKED dropped), as test_purge_skips_entries_claimed_by_concurrent_purger; - racing deletions of one collection queue one tombstone and leave the name free, and a deletion racing another process's waits and then queues nothing (fail with a read before the DELETE), as test_concurrent_partition_deletes_are_clean and test_concurrent_remote_delete_yields_single_queue_entry; - a mint colliding with a live incarnation is minted again, not reported as a taken name (fails with every integrity error read as a taken name), as test_incarnation_colliding_with_live_partition_is_never_reused; - a mint colliding with a deletion in flight checks the queue after its insert (fails with the check moved before it), as test_mint_detects_collision_with_concurrent_deletion and its SQLite counterpart, staged by the insert being issued rather than by a sleep; - a round claims one tombstone (fails without the LIMIT), as test_purge_claims_queue_entries_incrementally; - a persistent database error surfaces with its cause chained (fails with the cause dropped), as test_persistent_integrity_error_surfaces_with_cause. The existing test that an incarnation awaiting purge is never minted again, the equivalent of test_incarnation_with_garbage_left_is_never_reused, was verified the same way. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
…cess Milvus Lite does not arbitrate two creations of one collection racing in one process: the loser fails with "[Errno 17] File exists", which is not an already-exists error, so the logical creation raised. Every logical collection of a namespace and configuration shares its native collection, so two sessions created at once before it exists race on it (on main too, for different names; the per-name locks this PR removed had covered the same name). The store now takes a per-process lock per native collection around the existence check and the creation; the later creator finds the collection. Creation stays idempotent across processes as before. The contract gains a lifecycle churn test, the equivalent of MemMachine#1661's test_lifecycle_churn_completes_without_database_errors: concurrent create, open-or-create, open and delete of a few names raise nothing but the domain's own outcomes. It found this on Milvus Lite and fails there without the lock. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
…ifecycle for Equivalents of the segment store tests MemMachine#1661 adds, in the lifecycle contract both stores run (Qdrant in-memory, REST and gRPC; Milvus Lite), each verified to fail with the behavior removed and to pass with it: - a read checks the registry once, before it, and a write twice, around it (fails with a check after a read, or without the check after a write), as test_reads_check_liveness_inside_the_data_statement pins the segment store's registry round trips per operation; - a creation that fails at the native collection registers nothing, for create_collection and open_or_create_collection (fails with the registry written first), as test_unloadable_codec_config_commits_no_registry_row; - open-or-create gives up after a bounded number of lost races (fails without the bound), and creates again when the winner it lost to is gone (fails if that case raises), as the lost-race arm of test_persistent_mint_failure_raises_instead_of_looping; - deletion leaves the records for the purge (fails if deletion reclaims them itself), as test_delete_partition_touches_only_registry_and_queue: the purge test asserted the records unreachable but not still held. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Sep 25, 2026
validate_vector_store_name matched `^[a-z0-9_]+$`, and `$` also matches before a trailing newline, so "name\n" passed as a vector store name, which names the native Qdrant or Milvus collection and is part of the SQLite stores' table names. It uses fullmatch, as validate_identifier does for partition keys and MemMachine#1661 does for the segment store's. The rule had no test: valid names pass, and an empty name, an uppercase letter, a hyphen, a trailing newline and a name one byte over the bound are refused. The trailing-newline case fails with `match` restored. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
This was referenced Sep 28, 2026
edwinyyyu
added a commit
that referenced
this pull request
Sep 30, 2026
…1712) (#1713) * Build SQLite's context statements once per request, binding each seed's position SQLite's context reads run two statements per seed, and each was built per seed with the seed's walk position as literals. Building a statement and computing its cache key costs several times what SQLite takes to run it: 0.16 ms against 0.04 ms per statement on a 200,000-segment partition, for a filtered statement. Each direction's statement is now built once per request and run per seed with the position bound. With 20 seeds, 2 back and 4 forward, a request takes 11.7 ms instead of 15.3 ms unfiltered and 12.4 ms instead of 17.0 ms with a property filter matching 20% (p50, one client). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Take filtered context from a bounded window next to each seed (fixes #1712) A context read with a property filter selected, per seed and direction, the first matching rows in walk order: ORDER BY the walk with the filter in the WHERE clause and LIMIT the context size. The planner has no statistics for property predicates; it estimates about one match per seed, costs the ordered walk as reading the seed's whole side, and on PostgreSQL plans a bitmap over that side and a sort instead. The read then grew with the partition rather than with the context asked for. With a filter, the context now comes from the seed's next 1,024 segments in walk order, read with no filter, and the filter and the context size apply over those rows. The inner read has no filter to misestimate, so it stays an ordered index scan, and it stops once the context is found or the window ends. A match further than 1,024 segments from the seed is no longer context; the segment store ABC now allows that bound. Unfiltered reads keep their statement. SQLite's per-seed reads take the same window, so both dialects return the same context. Measured against #1661's tip, 20 seeds per request with 2 segments back and 4 forward, the rows of 21 partitions interleaved: - PostgreSQL 16: a filtered request takes 8-12 ms instead of 4.5-7.1 s on a 1,000,000-segment partition, and 8-13 ms instead of 103-115 ms on 20,000-segment ones; unfiltered, 7.6 ms against 7.8 ms. - SQLite, where the plan never read the whole side: filters matching 20% or 1% take 1.1-1.2x as long as with the previous commit alone (14.1 vs 12.4 ms, 28.4 vs 24.2 ms), while a property that only the seeds carry takes 36 ms instead of 602 ms on a 200,000-segment partition, since the walk no longer runs toward the partition's end. - For the same seeds, both dialects returned the same context as before, except where the bound applies: 12 forward with a filter matching every 100th segment returns the 10 matches within 1,024 segments. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Rename the context reads' ordering helper to _chronological_order The helper orders segment rows by their chronological columns (timestamp, event, index, offset), newest first when descending, the order the store's context reads called chronological_order before the window. _walk_order named no column or direction. The window query's docstring now says its result comes nearest the seed first, which holds on both sides. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Bound filtered context reads at 1,000 segments per side, not 1,024 Nothing depends on a power of two here: the bound is a LIMIT on the segments a filtered context read takes per seed and side, and 1,000 is the round number. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Build each context statement's walk in _context_rows_query Both context reads built the same walk before handing it to _context_rows_query: the partition's segments past the seed on one side, nearest first. _context_rows_query now takes the seed's ordering values and the side, and builds the walk itself. Its callers pass only what differs between them, bound parameters on SQLite and the seeds subquery's columns on PostgreSQL, and no longer pair a range condition with a direction flag that has to agree with it. The walk carries the partition's liveness check, so every context statement carries it from one place. PostgreSQL's statement had it in its seeds subquery, which drops it; otherwise its SQL is unchanged, and SQLite's is byte-identical. correlate_except(SegmentRow) replaces the lateral's two correlate(seeds_subquery) calls: only the walk needed one, when a property filter nests it in the window. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Shorten _context_rows_query's docstring to its contract The docstring now says what the statement selects, what the seed's ordering values may be, what the window bounds and why it is read without the filter, and that a deleted partition yields nothing. The account of how an unestimated filter turns the walk into a read of the seed's whole side stays in the PR and #1712. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Pass the context helpers' flags and bounds by keyword Calls like _context_rows_query(values, True, max_backward_segments, property_filter) did not say what True or the bound were. The side, limit and filter of _context_rows_query and of the lateral's per-direction helper, _chronological_order's descending, and _resolve_segment_field's row are now keyword-only; each helper's subject (the seed's ordering values, the row, the field) stays positional. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Say plainly why SQLite's context statements are built once per request The comment now names what is bound per seed (its timestamp, event UUID, index and offset) and why: building a statement costs more CPU than SQLite spends running it, so building one per seed would dominate the read. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Test context reads against a model of the timeline The existing context tests pass with the lateral's per-seed walk left uncorrelated to its seed (only one filtered test, and it has one seed) and with segments sharing a timestamp ordered by the wrong tie-breakers (every test gives each segment its own timestamp). The new test reads random contexts, several seeds at a time, with filters of every node type, from a partition where many segments share a timestamp and another partition holds segments at the same times, and compares each result with an in-memory model that evaluates filters in SQL's logic. Every partition is far smaller than the filtered read's window, so the model has no bound. It passes on both dialects here and at #1661's tip, and fails on both with the window ordered the same way on either side, with the filter resolved against the table instead of the window, and with the tie-breakers reordered, and on PostgreSQL with the walk uncorrelated. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Build context statements from Core columns Building the window's statement through an ORM alias adapted every column the filter and the ordering read from it, and the walk went through the mapped class. With a property filter that made building a PostgreSQL request's two statements, with SQLAlchemy's cache key, cost 1.6-1.8 ms of Python against 0.86 ms at #1661's tip. The walk and the window now use the table's and the window's own columns, and SQLite's per-seed statements load segment rows through select(SegmentRow).from_statement(), since building them from plain rows cost more than ORM loading. The SQL is byte-identical on both dialects. A filtered request now takes 1.15-1.21 ms to build, of which about 0.3 ms is the window itself: its subquery's columns and the larger statement to hash. Filtered reads on PostgreSQL partitions of 200 and 2,000 segments, 20 seeds, went from 4-28% slower than #1661's tip to 1-9%; SQLite reads are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Annotate column collections with the documented ColumnCollection ReadOnlyColumnCollection, what FromClause.c returns, is importable only from sqlalchemy.sql.base and has no entry in SQLAlchemy's documentation. ColumnCollection, its base class, is documented and exported from sqlalchemy. Annotations only. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Bound every context read at the same distance, filtered or not The SegmentStore ABC allowed a bound only on filtered context. It now allows an implementation-defined bound on how many of the segments nearest each seed context may come from, whatever the filter: a segment that far from its seed is not a neighbor either way, and a hard bound keeps every context read's cost bounded. The store applies its window to unfiltered reads too by capping their limit at it: an unfiltered read already reads exactly its context size, so the cap needs no window subquery and costs nothing. The constant is renamed _MAX_CONTEXT_DISTANCE, and the window's test now checks that an unfiltered read stops at the bound; it fails with the cap removed. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * State the context bound as the caller's contract The ABC now says what a caller can rely on, that each side's context may be limited to an implementation-defined number of the segments nearest the seed, rather than describing what an implementation does beyond the bound. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * State the context bound as what a caller can observe The ABC now says only that an implementation may bound how far from each seed it looks for context, so a side can come back with fewer segments than requested even when more exist further away. It no longer says how the bound is counted, so it neither reads as though context might not match property_filter nor rules out an implementation with no bound or with a bound on matching segments. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Build SQLite's context statements only for the sides a read asks for The loop built both directions' statements on every read and ran each only when its side asked for segments, so a read with no backward context (expand_context of 1 or 2) built a statement it never used. Each statement is now built only when its side asks for segments, as the PostgreSQL path already does. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Test the context bound with several seeds, filters and ties The bound test used one seed, and the model test's partitions never reached the bound, so no test exercised the bound with several seeds, filters and shared timestamps together. Each model-test read now sets the bound at random, from one segment to the default, and the model applies it before the filter: the nearest segments on each side, then the filter and the requested count. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Load PostgreSQL's context rows through the ORM The lateral path built each context row by hand, passing ten keyword arguments to SegmentRow's instrumented constructor, as main did. It now selects each seed's UUID next to SegmentRow aliased to the lateral subquery, so the ORM loads the rows, as the SQLite path does. The SQL selects the same columns plus incarnation, and the contexts are identical. Measured against the previous commit in one quiet window, arms alternated (20 seeds, 2 back and 4 forward unless noted): p50 at one client 14-22% lower with a filter on the 1M-segment partition and the 20k one with a 20% filter (7.6 to 5.9 ms with 20%; 22.2 to 19.0 ms with 1%, 4 back and 12 forward), and requests/s at 16 clients 26-39% higher in every scenario. Building a request's two statements takes 0.95-0.98 ms with a filter instead of 1.18-1.22 ms, and 0.46-0.48 ms without one instead of 0.74-0.78 ms, since the select list is one entity instead of ten columns. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Resolve filter fields against columns only _resolve_segment_field took either the mapped class or a column collection, defaulting to the class, because the seeds query passed the class and the window passed its columns. The seeds query now passes the table's columns too, so the resolver takes one kind of argument, named columns like _chronological_order's, with no default, and no longer needs .expression for the timestamp column. The compiled SQL is unchanged on both dialects. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Read SQLite's context one direction at a time, as PostgreSQL does The loop built and ran the backward and forward statements in two near-identical blocks each. It now has the PostgreSQL path's shape: a per-direction function that builds the direction's statement once and runs it for every seed, called for a side only when that side asks for segments. The statements are the same; all of a read's backward statements now run before its forward ones. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Revert "Test the context bound with several seeds, filters and ties" This reverts commit 745c679. The bound's size is not part of the contract, and bounds of a few segments are not realistic use. The bound test already fails with the window's limit off by one or removed and with the unfiltered cap removed, and the model test, without a bound, already fails on PostgreSQL with a seed's walk uncorrelated to its seed, the multi-seed risk the random bounds were meant to cover. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Draw the model test's identifiers from its seeded generator The model test's timestamps, properties, seeds and filters came from a seeded generator, but segment, event and derivative identifiers came from uuid4(), so which of two events sharing a timestamp came first changed from run to run. They now come from the same generator, so every run generates the same data. _seg takes an optional uuid for it. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Revert "Draw the model test's identifiers from its seeded generator" This reverts commit 9aac74f. The seeded generator already makes every run test the same cases: the timestamps, properties, seeds, context sizes and filters. Identifiers stay random UUIDs from uuid4(); they only decide which of two events sharing a timestamp sorts first, which the model orders the same way. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Give the model test's events sorted random UUIDs Events sharing a timestamp are ordered by their UUIDs, which came from uuid4() in generation order, so their order changed from run to run. The test now draws its events' UUIDs from uuid4(), sorts them, and hands them out in generation order, so every run tests the same cases, including the order of tied events, while the UUIDs stay random. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Oct 1, 2026
From an audit against the PR's writing rules, across the seven vector store design documents: - They narrated changes and PRs (what "is gone", what the configuration "gains" and "loses", what MemMachine#1627, MemMachine#1663, MemMachine#1625, MemMachine#1661 and MemMachine#1671 do, and "the previous design"), and claimed MemMachine#1468 fixed by two PRs that are open. Each now states the design, keeping links to open issues. - They had drifted from the code: a delete checks liveness once, after its call; Qdrant has no load step and Milvus loads on every preparation; the handle's writes run with qdrant-client's default wait rather than passing it; the Qdrant store states no replicated read delay; and the purge contract promises bounded work per call, while oldest-first rounds are the registry-backed stores'. - Repetition across documents goes: the clients section, the replicated Qdrant measurements and the Qdrant id alternatives now live in one document each, linked from the others; consistency cites the Strong wait's measurement instead of a figure without its conditions; the async-client measurement names its Milvus version. - The UUID text form, a string format rather than a design decision, is gone from the isolation document. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This was referenced Oct 1, 2026
Tracking: make the session lifecycle safe across replicas with a registry row and durable jobs
#1755
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port
Copy of #1548, merged into
speedkickas7752e4cb7, onmain: the same commit cherry-picked. Its added and removed lines equal 7752e4c's in every file butuv.lock, whose hunk is dropped (it also dropped staledatasets,dill,multiprocessandfsspec[http]entries that main's lock already lacks). Context differs in one file,test_event_backend_wiring.py;database_manager.py's surroundings match again since #1679 (the port of #1552) reached main. One line differs intest_resource_manager.py: the origin addsfrom unittest.mock import create_autospecwhere main's line already importsAsyncMockandpatch(#1683), so the port extends that line. A second commit drops a duplicated@overrideonpurge_deleted_partitionsthat the speedkick merge carried. #1548's Testing section is speedkick's run; the Verification below is this branch's.Purpose of the change
Addresses #1544, #1546, and #1549 by construction: the segment store's per-tenant partitioned layout is replaced with shared tables and an incarnation-carrying tenant registry, on every dialect. Design record:
design/segment_store_shared_tables.md.Also fixes #1557, #1558, and #1559: #1462's UTC-normalization fixes are folded in for every SQL store (segment, episode, cluster), so no store ships with the SQLite timestamp corruption.
Why an overhaul instead of the earlier targeted fixes
This PR began as three targeted repairs to the partitioned layout (detach-before-drop for #1544, a management lock for the create/delete deadlock, child-table DML for #1546), each verified on its own. Further investigation showed the layout itself was the root cause and could not be fully repaired:
Description
segment_store_ptis the tenant registry: logical key (primary key) plus anincarnation, a random unique-constrained UUID. Data rows carry the incarnation alone, so a data query cannot be built without resolving the registry, and referencing the wrong tenant is structurally impossible. A deleted-and-recreated tenant never sees its predecessor's rows, even mid-purge. A colliding mint is rejected rather than left to probability: the unique constraint rejects a collision with a live incarnation, and the mint transaction re-checks the purge queue after its insert so an incarnation whose garbage is still awaiting purge is re-minted (race-free with the existing tables: the check runs after the registry insert, and no new queue entry for the minted value can appear before commit, since the only registry row carrying it is uncommitted). Any integrity rejection with no committed row under the key is likewise retried with a fresh incarnation, up to_MAX_MINT_ATTEMPTS; a persistent cause surfaces asSegmentStoreAttemptsExhaustedErrorwith the database error chained. Rejections are warning-logged at the detection site, since a genuine collision is astronomically unlikely and the log is the signal for broken randomness or a persistent database error even when the re-mint heals it.create_partitionis a row insert (microseconds, no DDL).delete_partitiontakesFOR UPDATEon the registry row (waiting out writers'FOR SHAREpins), enqueues the incarnation onsegment_store_gc(logical key kept for forensics), and deletes the row: O(1) at any tenant size, the pool-model contract of "unreachable immediately, erased asynchronously".purge_deleted_partitions() -> boolis part of theSegmentStoreinterface. The caller's whole protocol is "call until False"; sizing is not a call argument, because callers cannot know engine-appropriate bounds. Deployments set them once at construction:purge_max_segments(default 10000) andpurge_max_partitions(default 100, its own bound because retiring an empty entry costs ~0.95 ms, ~200x a segment row, and empty partitions are cheap to mass-create-and-delete). Each call is one transaction (measured ~147k-312k rows/s), committing its progress or nothing. Within an entry, a call continues from the entry's cursor,purged_through(the last segment key purged): it reads the next keys after it in primary-key order, deletes that key range, and records its last key in the same transaction, so no batch rereads rows earlier batches deleted (PostgreSQL keeps them in the table and its indexes until vacuum), and the call that finds the entry empty costs one index lookup. The cursor relies on no row being written into an incarnation once it is enqueued, which the write fence guarantees. Links follow byON DELETE CASCADE, 50-68% faster than manual link deletion at one and four links per segment; cascade deletion saturates ~3M link rows/s (380k segments/s at 1 link/segment, 46k at 64), so heavily linked partitions purge in sub-second calls. Link fan-out is set by the deriver, which the store cannot reject after derivation. Link rows go by the cascade alone: the store requires foreign-key enforcement of its engine and does not verify it. Entries are claimed one at a time, oldest first (enqueue stamped by the database clock, indexed), withFOR UPDATE SKIP LOCKED, so a call neither materializes nor locks the rest of the backlog and concurrent purgers from any process share the queue; only the claiming call touches a dead incarnation's rows, so reclamation is deadlock-free by construction. On SQLite, which drops locking clauses and defers BEGIN to the first write, the claim is an UPDATE of the queue row, so purgers serialize at the claim and each reads the cursor its predecessor committed. The store never schedules purging; implementations whose deletes reclaim physically returnFalse. The delete path does not purge: an inline drain was tried, first of the global queue and then scoped to the deleted key, and removed, since prompt physical erasure is not a promise the store can keep on every dialect (on SQLite any writer past the busy timeout fails) and the sweeper reclaims in the background, at mostpurge_max_segmentsrows per call with busy calls a pause apart (under 10,000 rows/s per process at the defaults); a deployment must run the sweeper, and the ABC says so.FOR SHAREwith an incarnation predicate. Reads add the same predicate to their data statement as anEXISTS, so one statement checks liveness and reads; a read that finds no rows issues the registry check on its own, to tell an empty partition from a stale handle, and a windowed read ends with one registry read because its statements take separate snapshots. On SQLite the driver defers BEGIN until the first write, so a SELECT-only fence checks nothing; BEGIN IMMEDIATE would mean taking over transaction management of the caller's shared engine, so the fence is a self-checking registry-row UPDATE (same write lock, scoped to the transaction, match count as the staleness check), and deletion opens its transaction the same way. A handle that outlives its partition or its incarnation raisesSegmentStorePartitionHandleStaleErroron every dialect; SQLite previously had no stale-handle detection at all. Cost: none on single-statement reads that find rows; one registry round trip on reads that find none, and at the end of every windowed read.enable_sqlite_foreign_keys, applied by the DatabaseManager to its engines): a listener added later misses connections the shared engine already pooled, a pre-existing bug surfaced in review round 4. The store states the requirement on itsengineparameter and does not verify it, which is the reference practice for SQLite foreign keys; an engine without the pragma leaves link rows that outlive their segments, and nothing reclaims them, since the purge relies on the same cascade. The SQLite vector stores carry the same latent pattern; filed as SQLite per-connection state (foreign keys, sqlite-vec extension) is registered per-store on caller-supplied engines #1568.created_atand itsstart_time/end_timebounds, clusterlast_tsand pendingcreated_at. Without this, SQLite (whoseDateTime(timezone=True)discards tzinfo) read non-UTC values back shifted by their offset. Datetime filter values are normalized to UTC-aware instants atComparison/Inconstruction, the filter language's contract stated onFilterExpr(a value denotes an instant; naive means UTC), so every consumer receives instants and compilers only choose a representation; the SQL column leaf binds values as-is, and remaining per-backend normalizations are idempotent defenses, removable separately. Two consumers change by design: the Neo4j compiler'sdatetime.timestamp()read a naive value in the server's local zone and now receives instants, with its ISO-string branch parsing under the same rule (pinned under a non-UTC process zone); the in-memory short-term evaluator tags a naive stored metadata datetime as UTC at comparison time, since stored user data is not rewritten. The episode store's bounds spell the convention inline as two explicit steps,ensure_tz_aware(...).astimezone(UTC), kept unwrapped so the naive-means-UTC decision stays visible at each site. PostgreSQL is unaffected. On SQLite, rows the episode and cluster stores previously wrote from non-UTC-offset datetimes hold the local wall clock as if it were UTC and read back shifted before and after this change alike; the offset was never stored, so only new writes are correct, and a range query spanning the upgrade mixes the two conventions.Measurements
Layout comparison and scaled runs are in the design doc. Headlines (pgvector:pg16, one harness for all arms): tenant creation 0.006 ms vs 6-12 ms of per-tenant DDL; ingest and read parity or better at 40x2k and 1M-pair scales including a 500k-row tenant; O(1) delete at any size; churn 0 deadlocks vs 41-83.
Store-API ABAB against upstream main (3 interleaved rounds, one PG instance, medians, at the revision before the read fence was folded into the data statement): ingest +42% (11.1k vs 7.8k pairs/s),
delete_segments+20%, event/derivative lookups 10-17% faster, windowed context expansion 8% faster (5.9 vs 6.4 ms), lifecycle create+open+delete 4.4x faster (2.9 vs 12.8 ms/cycle); seed context reads +0.19 ms (1.30 vs 1.11 ms) from the fence's separate registry round trip, since removed. Read-path ABAB of that fold (5 interleaved rounds, medians): seed context reads 1.17 vs 1.42 ms, event lookups 1.04 vs 1.61 ms, derivative lookups 1.05 vs 1.33 ms, windowed context expansion 8.74 vs 9.67 ms (8 rounds x 600 reps, paired median -0.91 ms); reads that find nothing unchanged. Server-side (EXPLAIN ANALYZE, best of 30) theEXISTSconjunct plans as a one-time InitPlan costing ~3 us per statement, so the client-side gain is the round trip. Purge drains ~237k rows/s when a partition's rows lie together on disk (12,000-row partitions, as the benchmark loaded them). With tenants' rows interleaved on disk, as concurrent ingestion lays them out, each deleted segment and its links sit on different pages, and the link cascade is 60-80% of a batch at one link per segment, about what deleting the links in bulk costs without a foreign key. Against the purge before the cursor (batches ofuuid IN (SELECT ... LIMIT n)from the incarnation's first key, with a link check before retiring), on PostgreSQL 16 (128 MB shared buffers) with 200,000-row dead incarnations interleaved in a 3.8M-row table, the two alternating over 4 rounds under an interactive load, VACUUM and CHECKPOINT before each phase: a purge takes 9.1 vs 14.9 s (medians) and the call that retires the entry 4 vs 1,042 ms; concurrent reads run at 292 vs 308/s and writes at 67 vs 66/s during it, both within the rounds' spread (398 and 108/s idle); WAL per purge is 1.9 vs 1.8 GB; with an older snapshot held open, 5.3 vs 10.9 s. On a contiguous 12,000-row partition the cursor's extra statements cost ~8% (237k vs 257k rows/s). The store-API, read-path, and EventMemory end-to-end (ingest, queries, context expansion, mixed load) benchmarks show no difference between the two beyond run-to-run noise. Claim and mint-check statements measure 215/156 us.Compatibility
No migration from the partitioned layout is provided: the event backend is opt-in and pre-GA, and existing databases recreate their schema (see the design doc). The partition-key contract (
[a-z0-9_], max 32 bytes) is unchanged. The store supports PostgreSQL and SQLite, the two dialects the configuration can build and CI tests: the engine validator refuses any other dialect with theValueErrorits other engine checks raise.main's store sent other dialects down a generic path; this one does not work there (on MySQL 8.4 every purge fails with error 1235, since MySQL rejectsDELETE ... IN (SELECT ... LIMIT n), and create/delete churn deadlocks with error 1213, as it already did onmain; on SQL Server, SQLAlchemy drops every locking clause and rendersJOIN LATERAL, which T-SQL lacks; on Oracle, SQLAlchemy cannot render the JSON columns).Stack
23 PRs on
main: 5 independent ones, and three consecutive stages numbered on their own: the vector store scale-out, the SQLite store fixes and the vector store contract changes. A stacked PR's diff on GitHub is cumulative until the PRs under it merge.Independent PRs, each directly on
main; review and merge in any order:The vector store stages, consecutive: each stacked on the one below. [vector store scale-out 1/5] is directly on
mainand 2/5 on it alone; they do not depend on the independent PRs. From 3/5 up, the chain's history also carries the independent PRs' commits beneath it, since the later stages were written on top of them ([vector store contract 2/6] on #1624's session policy, for one); so a merge of an independent PR rebases the chain without conflict, and until they merge those PRs' changes show in a stacked PR's diff.speedkick, #1588)speedkick, #1589)This PR is its 14 commits,
8bff527a6,4b6522504,f007f6ae9,5d65f2508,bbacda759,1f6ab70a6,a5b2b853f,18d628cd9,d4a04523c,73f4e2b88,4efb7ac3c,6906a805c,7eff93fcc,c5ace6ace(the third drops the resource manager's closed gate, which guarded a get racingclose()that the server's shutdown order never produces; the fourth refuses engines of dialects other than PostgreSQL and SQLite; the fifth has the tests stamp queue entries with the database's clock and arithmetic, as the store does, instead of Python times; the sixth purges in primary-key ranges, and the seventh reverts it: with fewer than a batch of rows left, its DELETE had no key bound and read every already-deleted row of the incarnation, making a purge's final call slower, 5.0-7.5 s against 0.84-1.5 s at 250,000 rows; the eighth purges from a cursor on the queue entry and makes the SQLite claim a write; the ninth leaves links to the foreign key's cascade, dropping the link check; the tenth adds tests that segment writes are atomic and that mixed operations strand nothing; the eleventh has the SQLite claim touch the queue row's incarnation, as the store's other SQLite write claims touch their row's key; the twelfth keeps an entry whenever a batch deleted at least the budget, so a batch that ever deleted more than it read could not retire an entry with rows left; the thirteenth corrects the design doc's reclamation claims, which still had the purge reclaiming leaked links after the ninth and a drain faster than the sweeper's pacing allows, and drops the same pacing claim fromLongTermMemory, leaving that file as onmain; the fourteenth importsColumnElementfromsqlalchemy, where it is publicly exported, instead of the internalsqlalchemy.sql.elements), directly onmain, independent of the others: review and merge it in any order among the independent PRs (a later one touching the same file rebases). The stages' history carries the same change from [vector store scale-out 3/5] up, so their diffs include it until it merges.Verification
At its first 2 commits, on 2026-09-21, and at
273763688,c2503baf2,2c1305066and6845ad690, on 2026-09-22 and 2026-09-23:ruff checkandruff format --checkclean;ty checkclean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); the full server suite without integration tests passes (pytest packages/server/server_tests -m "not integration"), 1971 tests at78b5b36de, 1970 at273763688(the removed test is the one that pinned the gate) and 1971 atc2503baf2,2c1305066and6845ad690(the added test is the dialect check's, verified to fail without the check; the FIFO test, re-stamped by the database, still fails with the claim order reversed); the resource manager and segment store integration tests pass against PostgreSQL at6845ad690. Atbc03bdb7d,01572175aand09ba90592, on 2026-09-23:ruff check,ruff format --checkandty check(as CI runs it) clean; the full server suite without integration tests passes, 1978, 1977 (the removed test is the link check's) and 1984 tests; the segment store tests pass on SQLite and against PostgreSQL at09ba90592, 100 each. At14ae7b5c4:ruff check,ruff format --checkandty checkclean, 1984 tests without integration, and the SQLite segment store tests (100) pass. At7b014ac34:ruff check,ruff format --checkandty checkclean; the segment store tests pass on SQLite and against PostgreSQL, 100 each. At81eeb1dc4(documentation and a removed comment only):ruff checkandruff format --checkclean on the changed source file. The new tests were checked against deliberately broken stores: the racing-purgers test fails with the SELECT claim restored on SQLite (4 of 6 rows left, 3 of 3 runs), the partial-add test fails when an add's segments commit apart from its links, and the model test fails whendelete_segmentsdeletes only one of its segments. Those results were taken at the commits as they stood before the branch was rebased ontomain'sfa66fb612and re-signed on 2026-09-24; in order, the commits named there are now8bff527a6,4b6522504,f007f6ae9,5d65f2508,bbacda759,1f6ab70a6,a5b2b853f,18d628cd9,d4a04523c,73f4e2b88,4efb7ac3c,6906a805cand7eff93fcc. Atc5ace6ace, on 2026-09-25:ruff check,ruff format --checkandty check(as CI runs it) clean; the full server suite without integration tests passes, 1982 tests; the event memory, long-term memory and resource manager integration tests pass against PostgreSQL, 103 with 3 skipped.🤖 Generated with Claude Code
https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn