Skip to content

Overhaul segment store: shared tables with incarnation-scoped tenant keys (port of #1548) - #1661

Merged
edwinyyyu merged 14 commits into
MemMachine:mainfrom
edwinyyyu:port/segment-store-shared-tables-main
Sep 25, 2026
Merged

edwinyyyu merged 14 commits into
MemMachine:mainfrom
edwinyyyu:port/segment-store-shared-tables-main

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Port

Copy of #1548, merged into speedkick as 7752e4cb7, on main: the same commit cherry-picked. Its added and removed lines equal 7752e4c's in every file but uv.lock, whose hunk is dropped (it also dropped stale datasets, dill, multiprocess and fsspec[http] entries that main's lock already lacks). Context differs in one file, test_event_backend_wiring.py; database_manager.py's surroundings match again since #1679 (the port of #1552) reached main. One line differs in test_resource_manager.py: the origin adds from unittest.mock import create_autospec where main's line already imports AsyncMock and patch (#1683), so the port extends that line. A second commit drops a duplicated @override on purge_deleted_partitions that the speedkick merge carried. #1548's Testing section is speedkick's run; the Verification below is this branch's.

Purpose of the change

Addresses #1544, #1546, and #1549 by construction: the segment store's per-tenant partitioned layout is replaced with shared tables and an incarnation-carrying tenant registry, on every dialect. Design record: design/segment_store_shared_tables.md.

Also fixes #1557, #1558, and #1559: #1462's UTC-normalization fixes are folded in for every SQL store (segment, episode, cluster), so no store ships with the SQLite timestamp corruption.

Why an overhaul instead of the earlier targeted fixes

This PR began as three targeted repairs to the partitioned layout (detach-before-drop for #1544, a management lock for the create/delete deadlock, child-table DML for #1546), each verified on its own. Further investigation showed the layout itself was the root cause and could not be fully repaired:

  • PostgreSQL's FK integrity triggers still addressed the partitioned parents internally, so writes and deletes kept a per-backend lock spike proportional to partition count even after every client statement was fixed (10 -> 166 -> 10 relation locks across the trigger's generic-plan attempt).
  • Writer-vs-lifecycle-DDL deadlocks through the shared parents survived every lock discipline: 41-83 deadlocks per 20-second churn on upstream and on every intermediate fix, in two cycle shapes captured from the server log, one of them a lock-upgrade cycle inherent to dropping a child that carries a cloned FK.
  • Scaling requirements ruled out every per-tenant-table variant: tenant creation must stay cheap at 1e5-1e6 tenants (a per-tenant table costs ~5 catalog relations plus disk files), with 1e4-1e7 rows per tenant.

Description

  • Shared tables everywhere. The ORM models are the physical schema on every dialect. PostgreSQL partitioning, per-tenant DDL, detach machinery, the store-wide management lock, and the per-partition entity cache are removed. With no partitions there is no generic-plan lock fan-out, and with no lifecycle DDL there is nothing for the deadlock class to live in: the churn smoke that measured 41-83 deadlocks/20s on every partitioned build measures zero, with ~60x more write throughput and ~200x more lifecycle throughput.
  • Incarnation-scoped tenant identity. segment_store_pt is the tenant registry: logical key (primary key) plus an incarnation, a random unique-constrained UUID. Data rows carry the incarnation alone, so a data query cannot be built without resolving the registry, and referencing the wrong tenant is structurally impossible. A deleted-and-recreated tenant never sees its predecessor's rows, even mid-purge. A colliding mint is rejected rather than left to probability: the unique constraint rejects a collision with a live incarnation, and the mint transaction re-checks the purge queue after its insert so an incarnation whose garbage is still awaiting purge is re-minted (race-free with the existing tables: the check runs after the registry insert, and no new queue entry for the minted value can appear before commit, since the only registry row carrying it is uncommitted). Any integrity rejection with no committed row under the key is likewise retried with a fresh incarnation, up to _MAX_MINT_ATTEMPTS; a persistent cause surfaces as SegmentStoreAttemptsExhaustedError with the database error chained. Rejections are warning-logged at the detection site, since a genuine collision is astronomically unlikely and the log is the signal for broken randomness or a persistent database error even when the re-mint heals it.
  • O(1) deletion plus a purge queue. create_partition is a row insert (microseconds, no DDL). delete_partition takes FOR UPDATE on the registry row (waiting out writers' FOR SHARE pins), enqueues the incarnation on segment_store_gc (logical key kept for forensics), and deletes the row: O(1) at any tenant size, the pool-model contract of "unreachable immediately, erased asynchronously".
  • Purge as an ABC capability. purge_deleted_partitions() -> bool is part of the SegmentStore interface. The caller's whole protocol is "call until False"; sizing is not a call argument, because callers cannot know engine-appropriate bounds. Deployments set them once at construction: purge_max_segments (default 10000) and purge_max_partitions (default 100, its own bound because retiring an empty entry costs ~0.95 ms, ~200x a segment row, and empty partitions are cheap to mass-create-and-delete). Each call is one transaction (measured ~147k-312k rows/s), committing its progress or nothing. Within an entry, a call continues from the entry's cursor, purged_through (the last segment key purged): it reads the next keys after it in primary-key order, deletes that key range, and records its last key in the same transaction, so no batch rereads rows earlier batches deleted (PostgreSQL keeps them in the table and its indexes until vacuum), and the call that finds the entry empty costs one index lookup. The cursor relies on no row being written into an incarnation once it is enqueued, which the write fence guarantees. Links follow by ON DELETE CASCADE, 50-68% faster than manual link deletion at one and four links per segment; cascade deletion saturates ~3M link rows/s (380k segments/s at 1 link/segment, 46k at 64), so heavily linked partitions purge in sub-second calls. Link fan-out is set by the deriver, which the store cannot reject after derivation. Link rows go by the cascade alone: the store requires foreign-key enforcement of its engine and does not verify it. Entries are claimed one at a time, oldest first (enqueue stamped by the database clock, indexed), with FOR UPDATE SKIP LOCKED, so a call neither materializes nor locks the rest of the backlog and concurrent purgers from any process share the queue; only the claiming call touches a dead incarnation's rows, so reclamation is deadlock-free by construction. On SQLite, which drops locking clauses and defers BEGIN to the first write, the claim is an UPDATE of the queue row, so purgers serialize at the claim and each reads the cursor its predecessor committed. The store never schedules purging; implementations whose deletes reclaim physically return False. The delete path does not purge: an inline drain was tried, first of the global queue and then scoped to the deleted key, and removed, since prompt physical erasure is not a promise the store can keep on every dialect (on SQLite any writer past the busy timeout fails) and the sweeper reclaims in the background, at most purge_max_segments rows per call with busy calls a pause apart (under 10,000 rows/s per process at the defaults); a deployment must run the sweeper, and the ABC says so.
  • Fencing (Segment store partition handles are not fenced to a partition incarnation: stale handles silently operate on a recreated partition #1549). Writes pin the registry row under FOR SHARE with an incarnation predicate. Reads add the same predicate to their data statement as an EXISTS, so one statement checks liveness and reads; a read that finds no rows issues the registry check on its own, to tell an empty partition from a stale handle, and a windowed read ends with one registry read because its statements take separate snapshots. On SQLite the driver defers BEGIN until the first write, so a SELECT-only fence checks nothing; BEGIN IMMEDIATE would mean taking over transaction management of the caller's shared engine, so the fence is a self-checking registry-row UPDATE (same write lock, scoped to the transaction, match count as the staleness check), and deletion opens its transaction the same way. A handle that outlives its partition or its incarnation raises SegmentStorePartitionHandleStaleError on every dialect; SQLite previously had no stale-handle detection at all. Cost: none on single-statement reads that find rows; one registry round trip on reads that find none, and at the end of every windowed read.
  • Foreign keys. The segment table's FK to the registry is removed on purpose (registry and data rows decouple so deletion is O(1)); the link table's FK to the segment table and its cascade remain, enforced by a single-row indexed check with no partition fan-out. On SQLite, enforcement is per-connection state registered at ENGINE creation (enable_sqlite_foreign_keys, applied by the DatabaseManager to its engines): a listener added later misses connections the shared engine already pooled, a pre-existing bug surfaced in review round 4. The store states the requirement on its engine parameter and does not verify it, which is the reference practice for SQLite foreign keys; an engine without the pragma leaves link rows that outlive their segments, and nothing reclaims them, since the purge relies on the same cascade. The SQLite vector stores carry the same latent pattern; filed as SQLite per-connection state (foreign keys, sqlite-vec extension) is registered per-store on caller-supplied engines #1568.
  • Folded in: Fix timezone-aware datetime roundtrip on SQLite #1462's SQL-store fixes (fixes [Bug]: Segment store corrupts non-UTC timezone-aware timestamps on SQLite (write path + filter bounds) #1557, [Bug]: Episode store corrupts non-UTC timezone-aware created_at on SQLite (write path + time-range bounds) #1558, [Bug]: Cluster store corrupts non-UTC timezone-aware timestamps on SQLite (last_ts, pending created_at) #1559). App-supplied timezone-aware datetimes are normalized to UTC before persisting in every SQL store: segment timestamps, episode created_at and its start_time/end_time bounds, cluster last_ts and pending created_at. Without this, SQLite (whose DateTime(timezone=True) discards tzinfo) read non-UTC values back shifted by their offset. Datetime filter values are normalized to UTC-aware instants at Comparison/In construction, the filter language's contract stated on FilterExpr (a value denotes an instant; naive means UTC), so every consumer receives instants and compilers only choose a representation; the SQL column leaf binds values as-is, and remaining per-backend normalizations are idempotent defenses, removable separately. Two consumers change by design: the Neo4j compiler's datetime.timestamp() read a naive value in the server's local zone and now receives instants, with its ISO-string branch parsing under the same rule (pinned under a non-UTC process zone); the in-memory short-term evaluator tags a naive stored metadata datetime as UTC at comparison time, since stored user data is not rewritten. The episode store's bounds spell the convention inline as two explicit steps, ensure_tz_aware(...).astimezone(UTC), kept unwrapped so the naive-means-UTC decision stays visible at each site. PostgreSQL is unaffected. On SQLite, rows the episode and cluster stores previously wrote from non-UTC-offset datetimes hold the local wall clock as if it were UTC and read back shifted before and after this change alike; the offset was never stored, so only new writes are correct, and a range query spanning the upgrade mixes the two conventions.

Measurements

Layout comparison and scaled runs are in the design doc. Headlines (pgvector:pg16, one harness for all arms): tenant creation 0.006 ms vs 6-12 ms of per-tenant DDL; ingest and read parity or better at 40x2k and 1M-pair scales including a 500k-row tenant; O(1) delete at any size; churn 0 deadlocks vs 41-83.

Store-API ABAB against upstream main (3 interleaved rounds, one PG instance, medians, at the revision before the read fence was folded into the data statement): ingest +42% (11.1k vs 7.8k pairs/s), delete_segments +20%, event/derivative lookups 10-17% faster, windowed context expansion 8% faster (5.9 vs 6.4 ms), lifecycle create+open+delete 4.4x faster (2.9 vs 12.8 ms/cycle); seed context reads +0.19 ms (1.30 vs 1.11 ms) from the fence's separate registry round trip, since removed. Read-path ABAB of that fold (5 interleaved rounds, medians): seed context reads 1.17 vs 1.42 ms, event lookups 1.04 vs 1.61 ms, derivative lookups 1.05 vs 1.33 ms, windowed context expansion 8.74 vs 9.67 ms (8 rounds x 600 reps, paired median -0.91 ms); reads that find nothing unchanged. Server-side (EXPLAIN ANALYZE, best of 30) the EXISTS conjunct plans as a one-time InitPlan costing ~3 us per statement, so the client-side gain is the round trip. Purge drains ~237k rows/s when a partition's rows lie together on disk (12,000-row partitions, as the benchmark loaded them). With tenants' rows interleaved on disk, as concurrent ingestion lays them out, each deleted segment and its links sit on different pages, and the link cascade is 60-80% of a batch at one link per segment, about what deleting the links in bulk costs without a foreign key. Against the purge before the cursor (batches of uuid IN (SELECT ... LIMIT n) from the incarnation's first key, with a link check before retiring), on PostgreSQL 16 (128 MB shared buffers) with 200,000-row dead incarnations interleaved in a 3.8M-row table, the two alternating over 4 rounds under an interactive load, VACUUM and CHECKPOINT before each phase: a purge takes 9.1 vs 14.9 s (medians) and the call that retires the entry 4 vs 1,042 ms; concurrent reads run at 292 vs 308/s and writes at 67 vs 66/s during it, both within the rounds' spread (398 and 108/s idle); WAL per purge is 1.9 vs 1.8 GB; with an older snapshot held open, 5.3 vs 10.9 s. On a contiguous 12,000-row partition the cursor's extra statements cost ~8% (237k vs 257k rows/s). The store-API, read-path, and EventMemory end-to-end (ingest, queries, context expansion, mixed load) benchmarks show no difference between the two beyond run-to-run noise. Claim and mint-check statements measure 215/156 us.

Compatibility

No migration from the partitioned layout is provided: the event backend is opt-in and pre-GA, and existing databases recreate their schema (see the design doc). The partition-key contract ([a-z0-9_], max 32 bytes) is unchanged. The store supports PostgreSQL and SQLite, the two dialects the configuration can build and CI tests: the engine validator refuses any other dialect with the ValueError its other engine checks raise. main's store sent other dialects down a generic path; this one does not work there (on MySQL 8.4 every purge fails with error 1235, since MySQL rejects DELETE ... IN (SELECT ... LIMIT n), and create/delete churn deadlocks with error 1213, as it already did on main; on SQL Server, SQLAlchemy drops every locking clause and renders JOIN LATERAL, which T-SQL lacks; on Oracle, SQLAlchemy cannot render the JSON columns).

Stack

23 PRs on main: 5 independent ones, and three consecutive stages numbered on their own: the vector store scale-out, the SQLite store fixes and the vector store contract changes. A stacked PR's diff on GitHub is cumulative until the PRs under it merge.

Independent PRs, each directly on main; review and merge in any order:

# PR change
— #1541 fix(event-backend): make expand_context return timeline-neighbor episodes
— #1661 (this PR) Overhaul segment store: shared tables with incarnation-scoped tenant keys (port of #1548)
— #1630 Bound every request to a remote vector store by a configured timeout
— #1682 Hand the Qdrant vector store a metrics factory, so its tracker emits (port of #1532)
— #1624 Make no memory request create a project

The vector store stages, consecutive: each stacked on the one below. [vector store scale-out 1/5] is directly on main and 2/5 on it alone; they do not depend on the independent PRs. From 3/5 up, the chain's history also carries the independent PRs' commits beneath it, since the later stages were written on top of them ([vector store contract 2/6] on #1624's session policy, for one); so a merge of an independent PR rebases the chain without conflict, and until they merge those PRs' changes show in a stacked PR's diff.

# PR change
Stage 1, the vector store scale-out: what horizontal scalability without sharding requires.
[vector store scale-out 1/5] #1671 Remove custom sharding from the Qdrant store (port of #1654)
[vector store scale-out 2/5] #1631 Mint an incarnation per collection life in a SQL-arbitrated registry, so any process may serve any Qdrant or Milvus collection
[vector store scale-out 3/5] #1670 Remove per-project filterable properties (port of #1606)
[vector store scale-out 4/5] #1702 Keep user properties out of the vector store
[vector store scale-out 5/5] #1627 Make a vector store one collection, with string-keyed partitions
Stage 2, the SQLite store fixes, on [vector store scale-out 5/5]: re-derived on the one-collection store.
[sqlite store fixes 1/7] #1460 Publish vector index files atomically (but not durably) (as merged into speedkick, #1588)
[sqlite store fixes 2/7] #1469 Never reuse a row id in SQLiteVectorStore (as merged into speedkick, #1589)
[sqlite store fixes 3/7] #1672 Own the search engine's concurrency in the store, not in each engine (port of #1612)
[sqlite store fixes 4/7] #1673 Serialize a partition's writes so the engine sees them in order (port of #1607)
[sqlite store fixes 5/7] #1674 Refuse a pending row replay cannot honor, instead of dropping it (port of #1608)
[sqlite store fixes 6/7] #1675 Take SQLite's write lock at BEGIN, not at the first write (port of #1609)
[sqlite store fixes 7/7] #1676 Give every write a fresh row id, so a key names one version (port of #1610)
Stage 3, the vector store contract changes, on [sqlite store fixes 7/7].
[vector store contract 1/6] #1663 Answer with cosine scores and uuids, not vectors and stale properties, and require a vector on the record (port of #1598 and #1603)
[vector store contract 2/6] #1622 Create a session's storage with the session, never on a request
[vector store contract 3/6] #1625 Remove open-or-create and close from both stores
[vector store contract 4/6] #1618 Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization
[vector store contract 5/6] #1628 Make a vector store filter only on the properties it declares
[vector store contract 6/6] #1616 Close the filter union, and make negation the complement on every backend

This PR is its 14 commits, 8bff527a6, 4b6522504, f007f6ae9, 5d65f2508, bbacda759, 1f6ab70a6, a5b2b853f, 18d628cd9, d4a04523c, 73f4e2b88, 4efb7ac3c, 6906a805c, 7eff93fcc, c5ace6ace (the third drops the resource manager's closed gate, which guarded a get racing close() that the server's shutdown order never produces; the fourth refuses engines of dialects other than PostgreSQL and SQLite; the fifth has the tests stamp queue entries with the database's clock and arithmetic, as the store does, instead of Python times; the sixth purges in primary-key ranges, and the seventh reverts it: with fewer than a batch of rows left, its DELETE had no key bound and read every already-deleted row of the incarnation, making a purge's final call slower, 5.0-7.5 s against 0.84-1.5 s at 250,000 rows; the eighth purges from a cursor on the queue entry and makes the SQLite claim a write; the ninth leaves links to the foreign key's cascade, dropping the link check; the tenth adds tests that segment writes are atomic and that mixed operations strand nothing; the eleventh has the SQLite claim touch the queue row's incarnation, as the store's other SQLite write claims touch their row's key; the twelfth keeps an entry whenever a batch deleted at least the budget, so a batch that ever deleted more than it read could not retire an entry with rows left; the thirteenth corrects the design doc's reclamation claims, which still had the purge reclaiming leaked links after the ninth and a drain faster than the sweeper's pacing allows, and drops the same pacing claim from LongTermMemory, leaving that file as on main; the fourteenth imports ColumnElement from sqlalchemy, where it is publicly exported, instead of the internal sqlalchemy.sql.elements), directly on main, independent of the others: review and merge it in any order among the independent PRs (a later one touching the same file rebases). The stages' history carries the same change from [vector store scale-out 3/5] up, so their diffs include it until it merges.

Verification

At its first 2 commits, on 2026-09-21, and at 273763688, c2503baf2, 2c1305066 and 6845ad690, on 2026-09-22 and 2026-09-23: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); the full server suite without integration tests passes (pytest packages/server/server_tests -m "not integration"), 1971 tests at 78b5b36de, 1970 at 273763688 (the removed test is the one that pinned the gate) and 1971 at c2503baf2, 2c1305066 and 6845ad690 (the added test is the dialect check's, verified to fail without the check; the FIFO test, re-stamped by the database, still fails with the claim order reversed); the resource manager and segment store integration tests pass against PostgreSQL at 6845ad690. At bc03bdb7d, 01572175a and 09ba90592, on 2026-09-23: ruff check, ruff format --check and ty check (as CI runs it) clean; the full server suite without integration tests passes, 1978, 1977 (the removed test is the link check's) and 1984 tests; the segment store tests pass on SQLite and against PostgreSQL at 09ba90592, 100 each. At 14ae7b5c4: ruff check, ruff format --check and ty check clean, 1984 tests without integration, and the SQLite segment store tests (100) pass. At 7b014ac34: ruff check, ruff format --check and ty check clean; the segment store tests pass on SQLite and against PostgreSQL, 100 each. At 81eeb1dc4 (documentation and a removed comment only): ruff check and ruff format --check clean on the changed source file. The new tests were checked against deliberately broken stores: the racing-purgers test fails with the SELECT claim restored on SQLite (4 of 6 rows left, 3 of 3 runs), the partial-add test fails when an add's segments commit apart from its links, and the model test fails when delete_segments deletes only one of its segments. Those results were taken at the commits as they stood before the branch was rebased onto main's fa66fb612 and re-signed on 2026-09-24; in order, the commits named there are now 8bff527a6, 4b6522504, f007f6ae9, 5d65f2508, bbacda759, 1f6ab70a6, a5b2b853f, 18d628cd9, d4a04523c, 73f4e2b88, 4efb7ac3c, 6906a805c and 7eff93fcc. At c5ace6ace, on 2026-09-25: ruff check, ruff format --check and ty check (as CI runs it) clean; the full server suite without integration tests passes, 1982 tests; the event memory, long-term memory and resource manager integration tests pass against PostgreSQL, 103 with 3 skipped.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

This was referenced Sep 16, 2026
@edwinyyyu
edwinyyyu force-pushed the port/segment-store-shared-tables-main branch from ddd65aa to 9196973 Compare September 17, 2026 17:39
@edwinyyyu edwinyyyu changed the title [speedkick port 3/17] Overhaul segment store: shared tables with incarnation-scoped tenant keys (port of #1548) [vector store 2/16] Overhaul segment store: shared tables with incarnation-scoped tenant keys (port of #1548) Sep 17, 2026
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
The existing context tests pass with the lateral's per-seed walk left
uncorrelated to its seed (only one filtered test, and it has one seed)
and with segments sharing a timestamp ordered by the wrong tie-breakers
(every test gives each segment its own timestamp). The new test reads
random contexts, several seeds at a time, with filters of every node
type, from a partition where many segments share a timestamp and
another partition holds segments at the same times, and compares each
result with an in-memory model that evaluates filters in SQL's logic.
Every partition is far smaller than the filtered read's window, so the
model has no bound.

It passes on both dialects here and at MemMachine#1661's tip, and fails on both
with the window ordered the same way on either side, with the filter
resolved against the table instead of the window, and with the
tie-breakers reordered, and on PostgreSQL with the walk uncorrelated.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
Building the window's statement through an ORM alias adapted every
column the filter and the ordering read from it, and the walk went
through the mapped class. With a property filter that made building a
PostgreSQL request's two statements, with SQLAlchemy's cache key,
cost 1.6-1.8 ms of Python against 0.86 ms at MemMachine#1661's tip. The walk and
the window now use the table's and the window's own columns, and
SQLite's per-seed statements load segment rows through
select(SegmentRow).from_statement(), since building them from plain
rows cost more than ORM loading.

The SQL is byte-identical on both dialects. A filtered request now
takes 1.15-1.21 ms to build, of which about 0.3 ms is the window
itself: its subquery's columns and the larger statement to hash.
Filtered reads on PostgreSQL partitions of 200 and 2,000 segments,
20 seeds, went from 4-28% slower than MemMachine#1661's tip to 1-9%; SQLite
reads are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
sqlalchemy.sql.elements is an internal module; the same class is
exported from sqlalchemy and documented as
sqlalchemy.sql.expression.ColumnElement. Same object, import path only.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@edwinyyyu
edwinyyyu merged commit 66d4a65 into MemMachine:main Sep 25, 2026
44 checks passed
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
…emMachine#1712)

A context read with a property filter selected, per seed and direction,
the first matching rows in walk order: ORDER BY the walk with the filter
in the WHERE clause and LIMIT the context size. The planner has no
statistics for property predicates; it estimates about one match per
seed, costs the ordered walk as reading the seed's whole side, and on
PostgreSQL plans a bitmap over that side and a sort instead. The read
then grew with the partition rather than with the context asked for.

With a filter, the context now comes from the seed's next 1,024
segments in walk order, read with no filter, and the filter and the
context size apply over those rows. The inner read has no filter to
misestimate, so it stays an ordered index scan, and it stops once the
context is found or the window ends. A match further than 1,024
segments from the seed is no longer context; the segment store ABC now
allows that bound. Unfiltered reads keep their statement. SQLite's
per-seed reads take the same window, so both dialects return the same
context.

Measured against MemMachine#1661's tip, 20 seeds per request with 2 segments back
and 4 forward, the rows of 21 partitions interleaved:

- PostgreSQL 16: a filtered request takes 8-12 ms instead of 4.5-7.1 s
  on a 1,000,000-segment partition, and 8-13 ms instead of 103-115 ms
  on 20,000-segment ones; unfiltered, 7.6 ms against 7.8 ms.
- SQLite, where the plan never read the whole side: filters matching
  20% or 1% take 1.1-1.2x as long as with the previous commit alone
  (14.1 vs 12.4 ms, 28.4 vs 24.2 ms), while a property that only the
  seeds carry takes 36 ms instead of 602 ms on a 200,000-segment
  partition, since the walk no longer runs toward the partition's end.
- For the same seeds, both dialects returned the same context as
  before, except where the bound applies: 12 forward with a filter
  matching every 100th segment returns the 10 matches within 1,024
  segments.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
The existing context tests pass with the lateral's per-seed walk left
uncorrelated to its seed (only one filtered test, and it has one seed)
and with segments sharing a timestamp ordered by the wrong tie-breakers
(every test gives each segment its own timestamp). The new test reads
random contexts, several seeds at a time, with filters of every node
type, from a partition where many segments share a timestamp and
another partition holds segments at the same times, and compares each
result with an in-memory model that evaluates filters in SQL's logic.
Every partition is far smaller than the filtered read's window, so the
model has no bound.

It passes on both dialects here and at MemMachine#1661's tip, and fails on both
with the window ordered the same way on either side, with the filter
resolved against the table instead of the window, and with the
tie-breakers reordered, and on PostgreSQL with the walk uncorrelated.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
Building the window's statement through an ORM alias adapted every
column the filter and the ordering read from it, and the walk went
through the mapped class. With a property filter that made building a
PostgreSQL request's two statements, with SQLAlchemy's cache key,
cost 1.6-1.8 ms of Python against 0.86 ms at MemMachine#1661's tip. The walk and
the window now use the table's and the window's own columns, and
SQLite's per-seed statements load segment rows through
select(SegmentRow).from_statement(), since building them from plain
rows cost more than ORM loading.

The SQL is byte-identical on both dialects. A filtered request now
takes 1.15-1.21 ms to build, of which about 0.3 ms is the window
itself: its subquery's columns and the larger statement to hash.
Filtered reads on PostgreSQL partitions of 200 and 2,000 segments,
20 seeds, went from 4-28% slower than MemMachine#1661's tip to 1-9%; SQLite
reads are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
validate_identifier matched `^[a-z0-9_]+$`, and `$` also matches before a
trailing newline, so "name\n" passed as a namespace, a collection name, and now
a vector store name that becomes part of the registry's table names. It uses
fullmatch, as MemMachine#1661 does for the segment store's partition keys. The event
backend's service locator already hashes such a session id: it checks session
ids with the segment store's validator, which uses fullmatch since MemMachine#1661.

Tests, each verified to fail with `match` restored: a vector store name with a
trailing newline is refused, and a namespace or collection name with one is
refused by both stores.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
The registry is built and tested on PostgreSQL and SQLite only. On
another dialect it misbehaves rather than failing at construction: the
retention cutoff is PostgreSQL's interval arithmetic outside SQLite,
MySQL has no DELETE ... RETURNING, and on SQL Server SQLAlchemy drops
the claims' FOR UPDATE SKIP LOCKED without a word. The constructor now
refuses any other dialect with a ValueError, beside its vector store
name check, as MemMachine#1661 does in the segment store's engine validator.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
…registry for

Equivalents of the segment store tests MemMachine#1661 adds, where the registry has the
same behavior, each verified to fail with that behavior removed and to pass
with it:
- a claim skips a tombstone another purger holds (PostgreSQL; fails with
  SKIP LOCKED dropped), as test_purge_skips_entries_claimed_by_concurrent_purger;
- racing deletions of one collection queue one tombstone and leave the name
  free, and a deletion racing another process's waits and then queues nothing
  (fail with a read before the DELETE), as test_concurrent_partition_deletes_are_clean
  and test_concurrent_remote_delete_yields_single_queue_entry;
- a mint colliding with a live incarnation is minted again, not reported as a
  taken name (fails with every integrity error read as a taken name), as
  test_incarnation_colliding_with_live_partition_is_never_reused;
- a mint colliding with a deletion in flight checks the queue after its insert
  (fails with the check moved before it), as
  test_mint_detects_collision_with_concurrent_deletion and its SQLite
  counterpart, staged by the insert being issued rather than by a sleep;
- a round claims one tombstone (fails without the LIMIT), as
  test_purge_claims_queue_entries_incrementally;
- a persistent database error surfaces with its cause chained (fails with the
  cause dropped), as test_persistent_integrity_error_surfaces_with_cause.
The existing test that an incarnation awaiting purge is never minted again, the
equivalent of test_incarnation_with_garbage_left_is_never_reused, was verified
the same way.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
…cess

Milvus Lite does not arbitrate two creations of one collection racing in one
process: the loser fails with "[Errno 17] File exists", which is not an
already-exists error, so the logical creation raised. Every logical collection
of a namespace and configuration shares its native collection, so two sessions
created at once before it exists race on it (on main too, for different
names; the per-name locks this PR removed had covered the same name). The
store now takes a per-process lock per native collection around the existence
check and the creation; the later creator finds the collection. Creation stays
idempotent across processes as before.

The contract gains a lifecycle churn test, the equivalent of MemMachine#1661's
test_lifecycle_churn_completes_without_database_errors: concurrent create,
open-or-create, open and delete of a few names raise nothing but the domain's
own outcomes. It found this on Milvus Lite and fails there without the lock.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
…ifecycle for

Equivalents of the segment store tests MemMachine#1661 adds, in the lifecycle contract
both stores run (Qdrant in-memory, REST and gRPC; Milvus Lite), each verified
to fail with the behavior removed and to pass with it:
- a read checks the registry once, before it, and a write twice, around it
  (fails with a check after a read, or without the check after a write), as
  test_reads_check_liveness_inside_the_data_statement pins the segment store's
  registry round trips per operation;
- a creation that fails at the native collection registers nothing, for
  create_collection and open_or_create_collection (fails with the registry
  written first), as test_unloadable_codec_config_commits_no_registry_row;
- open-or-create gives up after a bounded number of lost races (fails without
  the bound), and creates again when the winner it lost to is gone (fails if
  that case raises), as the lost-race arm of
  test_persistent_mint_failure_raises_instead_of_looping;
- deletion leaves the records for the purge (fails if deletion reclaims them
  itself), as test_delete_partition_touches_only_registry_and_queue: the purge
  test asserted the records unreachable but not still held.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
validate_identifier matched `^[a-z0-9_]+$`, and `$` also matches before a
trailing newline, so "name\n" passed as a namespace, a collection name, and now
a vector store name that becomes part of the registry's table names. It uses
fullmatch, as MemMachine#1661 does for the segment store's partition keys. The event
backend's service locator already hashes such a session id: it checks session
ids with the segment store's validator, which uses fullmatch since MemMachine#1661.

Tests, each verified to fail with `match` restored: a vector store name with a
trailing newline is refused, and a namespace or collection name with one is
refused by both stores.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
The registry is built and tested on PostgreSQL and SQLite only. On
another dialect it misbehaves rather than failing at construction: the
retention cutoff is PostgreSQL's interval arithmetic outside SQLite,
MySQL has no DELETE ... RETURNING, and on SQL Server SQLAlchemy drops
the claims' FOR UPDATE SKIP LOCKED without a word. The constructor now
refuses any other dialect with a ValueError, beside its vector store
name check, as MemMachine#1661 does in the segment store's engine validator.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
…registry for

Equivalents of the segment store tests MemMachine#1661 adds, where the registry has the
same behavior, each verified to fail with that behavior removed and to pass
with it:
- a claim skips a tombstone another purger holds (PostgreSQL; fails with
  SKIP LOCKED dropped), as test_purge_skips_entries_claimed_by_concurrent_purger;
- racing deletions of one collection queue one tombstone and leave the name
  free, and a deletion racing another process's waits and then queues nothing
  (fail with a read before the DELETE), as test_concurrent_partition_deletes_are_clean
  and test_concurrent_remote_delete_yields_single_queue_entry;
- a mint colliding with a live incarnation is minted again, not reported as a
  taken name (fails with every integrity error read as a taken name), as
  test_incarnation_colliding_with_live_partition_is_never_reused;
- a mint colliding with a deletion in flight checks the queue after its insert
  (fails with the check moved before it), as
  test_mint_detects_collision_with_concurrent_deletion and its SQLite
  counterpart, staged by the insert being issued rather than by a sleep;
- a round claims one tombstone (fails without the LIMIT), as
  test_purge_claims_queue_entries_incrementally;
- a persistent database error surfaces with its cause chained (fails with the
  cause dropped), as test_persistent_integrity_error_surfaces_with_cause.
The existing test that an incarnation awaiting purge is never minted again, the
equivalent of test_incarnation_with_garbage_left_is_never_reused, was verified
the same way.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
…cess

Milvus Lite does not arbitrate two creations of one collection racing in one
process: the loser fails with "[Errno 17] File exists", which is not an
already-exists error, so the logical creation raised. Every logical collection
of a namespace and configuration shares its native collection, so two sessions
created at once before it exists race on it (on main too, for different
names; the per-name locks this PR removed had covered the same name). The
store now takes a per-process lock per native collection around the existence
check and the creation; the later creator finds the collection. Creation stays
idempotent across processes as before.

The contract gains a lifecycle churn test, the equivalent of MemMachine#1661's
test_lifecycle_churn_completes_without_database_errors: concurrent create,
open-or-create, open and delete of a few names raise nothing but the domain's
own outcomes. It found this on Milvus Lite and fails there without the lock.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
…ifecycle for

Equivalents of the segment store tests MemMachine#1661 adds, in the lifecycle contract
both stores run (Qdrant in-memory, REST and gRPC; Milvus Lite), each verified
to fail with the behavior removed and to pass with it:
- a read checks the registry once, before it, and a write twice, around it
  (fails with a check after a read, or without the check after a write), as
  test_reads_check_liveness_inside_the_data_statement pins the segment store's
  registry round trips per operation;
- a creation that fails at the native collection registers nothing, for
  create_collection and open_or_create_collection (fails with the registry
  written first), as test_unloadable_codec_config_commits_no_registry_row;
- open-or-create gives up after a bounded number of lost races (fails without
  the bound), and creates again when the winner it lost to is gone (fails if
  that case raises), as the lost-race arm of
  test_persistent_mint_failure_raises_instead_of_looping;
- deletion leaves the records for the purge (fails if deletion reclaims them
  itself), as test_delete_partition_touches_only_registry_and_queue: the purge
  test asserted the records unreachable but not still held.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
validate_vector_store_name matched `^[a-z0-9_]+$`, and `$` also matches
before a trailing newline, so "name\n" passed as a vector store name, which
names the native Qdrant or Milvus collection and is part of the SQLite
stores' table names. It uses fullmatch, as validate_identifier does for
partition keys and MemMachine#1661 does for the segment store's.

The rule had no test: valid names pass, and an empty name, an uppercase
letter, a hyphen, a trailing newline and a name one byte over the bound
are refused. The trailing-newline case fails with `match` restored.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
edwinyyyu added a commit that referenced this pull request Sep 30, 2026
…1712) (#1713)

* Build SQLite's context statements once per request, binding each seed's position

SQLite's context reads run two statements per seed, and each was built
per seed with the seed's walk position as literals. Building a
statement and computing its cache key costs several times what SQLite
takes to run it: 0.16 ms against 0.04 ms per statement on a
200,000-segment partition, for a filtered statement. Each direction's
statement is now built once per request and run per seed with the
position bound. With 20 seeds, 2 back and 4 forward, a request takes
11.7 ms instead of 15.3 ms unfiltered and 12.4 ms instead of 17.0 ms
with a property filter matching 20% (p50, one client).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Take filtered context from a bounded window next to each seed (fixes #1712)

A context read with a property filter selected, per seed and direction,
the first matching rows in walk order: ORDER BY the walk with the filter
in the WHERE clause and LIMIT the context size. The planner has no
statistics for property predicates; it estimates about one match per
seed, costs the ordered walk as reading the seed's whole side, and on
PostgreSQL plans a bitmap over that side and a sort instead. The read
then grew with the partition rather than with the context asked for.

With a filter, the context now comes from the seed's next 1,024
segments in walk order, read with no filter, and the filter and the
context size apply over those rows. The inner read has no filter to
misestimate, so it stays an ordered index scan, and it stops once the
context is found or the window ends. A match further than 1,024
segments from the seed is no longer context; the segment store ABC now
allows that bound. Unfiltered reads keep their statement. SQLite's
per-seed reads take the same window, so both dialects return the same
context.

Measured against #1661's tip, 20 seeds per request with 2 segments back
and 4 forward, the rows of 21 partitions interleaved:

- PostgreSQL 16: a filtered request takes 8-12 ms instead of 4.5-7.1 s
  on a 1,000,000-segment partition, and 8-13 ms instead of 103-115 ms
  on 20,000-segment ones; unfiltered, 7.6 ms against 7.8 ms.
- SQLite, where the plan never read the whole side: filters matching
  20% or 1% take 1.1-1.2x as long as with the previous commit alone
  (14.1 vs 12.4 ms, 28.4 vs 24.2 ms), while a property that only the
  seeds carry takes 36 ms instead of 602 ms on a 200,000-segment
  partition, since the walk no longer runs toward the partition's end.
- For the same seeds, both dialects returned the same context as
  before, except where the bound applies: 12 forward with a filter
  matching every 100th segment returns the 10 matches within 1,024
  segments.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Rename the context reads' ordering helper to _chronological_order

The helper orders segment rows by their chronological columns
(timestamp, event, index, offset), newest first when descending, the
order the store's context reads called chronological_order before
the window. _walk_order named no column or direction. The window
query's docstring now says its result comes nearest the seed first,
which holds on both sides.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Bound filtered context reads at 1,000 segments per side, not 1,024

Nothing depends on a power of two here: the bound is a LIMIT on the
segments a filtered context read takes per seed and side, and 1,000
is the round number.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Build each context statement's walk in _context_rows_query

Both context reads built the same walk before handing it to
_context_rows_query: the partition's segments past the seed on one
side, nearest first. _context_rows_query now takes the seed's ordering
values and the side, and builds the walk itself. Its callers pass only
what differs between them, bound parameters on SQLite and the seeds
subquery's columns on PostgreSQL, and no longer pair a range condition
with a direction flag that has to agree with it.

The walk carries the partition's liveness check, so every context
statement carries it from one place. PostgreSQL's statement had it in
its seeds subquery, which drops it; otherwise its SQL is unchanged, and
SQLite's is byte-identical. correlate_except(SegmentRow) replaces the
lateral's two correlate(seeds_subquery) calls: only the walk needed
one, when a property filter nests it in the window.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Shorten _context_rows_query's docstring to its contract

The docstring now says what the statement selects, what the seed's
ordering values may be, what the window bounds and why it is read
without the filter, and that a deleted partition yields nothing. The
account of how an unestimated filter turns the walk into a read of the
seed's whole side stays in the PR and #1712.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Pass the context helpers' flags and bounds by keyword

Calls like _context_rows_query(values, True, max_backward_segments,
property_filter) did not say what True or the bound were. The side,
limit and filter of _context_rows_query and of the lateral's
per-direction helper, _chronological_order's descending, and
_resolve_segment_field's row are now keyword-only; each helper's
subject (the seed's ordering values, the row, the field) stays
positional.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Say plainly why SQLite's context statements are built once per request

The comment now names what is bound per seed (its timestamp, event
UUID, index and offset) and why: building a statement costs more CPU
than SQLite spends running it, so building one per seed would dominate
the read.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Test context reads against a model of the timeline

The existing context tests pass with the lateral's per-seed walk left
uncorrelated to its seed (only one filtered test, and it has one seed)
and with segments sharing a timestamp ordered by the wrong tie-breakers
(every test gives each segment its own timestamp). The new test reads
random contexts, several seeds at a time, with filters of every node
type, from a partition where many segments share a timestamp and
another partition holds segments at the same times, and compares each
result with an in-memory model that evaluates filters in SQL's logic.
Every partition is far smaller than the filtered read's window, so the
model has no bound.

It passes on both dialects here and at #1661's tip, and fails on both
with the window ordered the same way on either side, with the filter
resolved against the table instead of the window, and with the
tie-breakers reordered, and on PostgreSQL with the walk uncorrelated.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Build context statements from Core columns

Building the window's statement through an ORM alias adapted every
column the filter and the ordering read from it, and the walk went
through the mapped class. With a property filter that made building a
PostgreSQL request's two statements, with SQLAlchemy's cache key,
cost 1.6-1.8 ms of Python against 0.86 ms at #1661's tip. The walk and
the window now use the table's and the window's own columns, and
SQLite's per-seed statements load segment rows through
select(SegmentRow).from_statement(), since building them from plain
rows cost more than ORM loading.

The SQL is byte-identical on both dialects. A filtered request now
takes 1.15-1.21 ms to build, of which about 0.3 ms is the window
itself: its subquery's columns and the larger statement to hash.
Filtered reads on PostgreSQL partitions of 200 and 2,000 segments,
20 seeds, went from 4-28% slower than #1661's tip to 1-9%; SQLite
reads are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Annotate column collections with the documented ColumnCollection

ReadOnlyColumnCollection, what FromClause.c returns, is importable only
from sqlalchemy.sql.base and has no entry in SQLAlchemy's documentation.
ColumnCollection, its base class, is documented and exported from
sqlalchemy. Annotations only.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Bound every context read at the same distance, filtered or not

The SegmentStore ABC allowed a bound only on filtered context. It now
allows an implementation-defined bound on how many of the segments
nearest each seed context may come from, whatever the filter: a segment
that far from its seed is not a neighbor either way, and a hard bound
keeps every context read's cost bounded.

The store applies its window to unfiltered reads too by capping their
limit at it: an unfiltered read already reads exactly its context size,
so the cap needs no window subquery and costs nothing. The constant is
renamed _MAX_CONTEXT_DISTANCE, and the window's test now checks that an
unfiltered read stops at the bound; it fails with the cap removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* State the context bound as the caller's contract

The ABC now says what a caller can rely on, that each side's context
may be limited to an implementation-defined number of the segments
nearest the seed, rather than describing what an implementation does
beyond the bound.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* State the context bound as what a caller can observe

The ABC now says only that an implementation may bound how far from
each seed it looks for context, so a side can come back with fewer
segments than requested even when more exist further away. It no longer
says how the bound is counted, so it neither reads as though context
might not match property_filter nor rules out an implementation with no
bound or with a bound on matching segments.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Build SQLite's context statements only for the sides a read asks for

The loop built both directions' statements on every read and ran each
only when its side asked for segments, so a read with no backward
context (expand_context of 1 or 2) built a statement it never used.
Each statement is now built only when its side asks for segments, as
the PostgreSQL path already does.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Test the context bound with several seeds, filters and ties

The bound test used one seed, and the model test's partitions never
reached the bound, so no test exercised the bound with several seeds,
filters and shared timestamps together. Each model-test read now sets
the bound at random, from one segment to the default, and the model
applies it before the filter: the nearest segments on each side, then
the filter and the requested count.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Load PostgreSQL's context rows through the ORM

The lateral path built each context row by hand, passing ten keyword
arguments to SegmentRow's instrumented constructor, as main did. It now
selects each seed's UUID next to SegmentRow aliased to the lateral
subquery, so the ORM loads the rows, as the SQLite path does. The SQL
selects the same columns plus incarnation, and the contexts are
identical.

Measured against the previous commit in one quiet window, arms
alternated (20 seeds, 2 back and 4 forward unless noted): p50 at one
client 14-22% lower with a filter on the 1M-segment partition and the
20k one with a 20% filter (7.6 to 5.9 ms with 20%; 22.2 to 19.0 ms
with 1%, 4 back and 12 forward), and requests/s at 16 clients 26-39%
higher in every scenario. Building a request's two statements takes
0.95-0.98 ms with a filter instead of 1.18-1.22 ms, and 0.46-0.48 ms
without one instead of 0.74-0.78 ms, since the select list is one
entity instead of ten columns.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Resolve filter fields against columns only

_resolve_segment_field took either the mapped class or a column
collection, defaulting to the class, because the seeds query passed the
class and the window passed its columns. The seeds query now passes the
table's columns too, so the resolver takes one kind of argument, named
columns like _chronological_order's, with no default, and no longer
needs .expression for the timestamp column. The compiled SQL is
unchanged on both dialects.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Read SQLite's context one direction at a time, as PostgreSQL does

The loop built and ran the backward and forward statements in two
near-identical blocks each. It now has the PostgreSQL path's shape: a
per-direction function that builds the direction's statement once and
runs it for every seed, called for a side only when that side asks for
segments. The statements are the same; all of a read's backward
statements now run before its forward ones.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Revert "Test the context bound with several seeds, filters and ties"

This reverts commit 745c679. The bound's size is not part of the
contract, and bounds of a few segments are not realistic use. The
bound test already fails with the window's limit off by one or removed
and with the unfiltered cap removed, and the model test, without a
bound, already fails on PostgreSQL with a seed's walk uncorrelated to
its seed, the multi-seed risk the random bounds were meant to cover.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Draw the model test's identifiers from its seeded generator

The model test's timestamps, properties, seeds and filters came from a
seeded generator, but segment, event and derivative identifiers came
from uuid4(), so which of two events sharing a timestamp came first
changed from run to run. They now come from the same generator, so
every run generates the same data. _seg takes an optional uuid for it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Revert "Draw the model test's identifiers from its seeded generator"

This reverts commit 9aac74f. The seeded generator already makes every
run test the same cases: the timestamps, properties, seeds, context
sizes and filters. Identifiers stay random UUIDs from uuid4(); they
only decide which of two events sharing a timestamp sorts first, which
the model orders the same way.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Give the model test's events sorted random UUIDs

Events sharing a timestamp are ordered by their UUIDs, which came from
uuid4() in generation order, so their order changed from run to run.
The test now draws its events' UUIDs from uuid4(), sorts them, and
hands them out in generation order, so every run tests the same cases,
including the order of tied events, while the UUIDs stay random.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Oct 1, 2026
From an audit against the PR's writing rules, across the seven vector
store design documents:
- They narrated changes and PRs (what "is gone", what the configuration
  "gains" and "loses", what MemMachine#1627, MemMachine#1663, MemMachine#1625, MemMachine#1661 and MemMachine#1671 do, and
  "the previous design"), and claimed MemMachine#1468 fixed by two PRs that are
  open. Each now states the design, keeping links to open issues.
- They had drifted from the code: a delete checks liveness once, after
  its call; Qdrant has no load step and Milvus loads on every
  preparation; the handle's writes run with qdrant-client's default
  wait rather than passing it; the Qdrant store states no replicated
  read delay; and the purge contract promises bounded work per call,
  while oldest-first rounds are the registry-backed stores'.
- Repetition across documents goes: the clients section, the replicated
  Qdrant measurements and the Qdrant id alternatives now live in one
  document each, linked from the others; consistency cites the Strong
  wait's measurement instead of a figure without its conditions; the
  async-client measurement names its Milvus version.
- The UUID text form, a string format rather than a design decision,
  is gone from the isolation document.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants