Add session, source and expansion to EventMemory (speedkick) - #1597

Implements the session/source/expansion part of design/event_memory_handoff.md (branch design/tenant-lifecycle, PR #1579): the accepted parts of the server redesign that live in EventMemory, SegmentStore and their data models. Eviction is split out into a second PR stacked on this one, and the context-part and block-kind model and the segmenter/deriver tables into a third stacked on that; here the producer/null context union and the block_type discriminator stay as they are. Nothing is renamed; nothing in the server is rewired beyond calling the new API.

Sits directly on speedkick (rebased 2026-09-14 onto a8322a7, after [vector store 1/13] #1606 and the SQLite store stack #1612, #1607-#1610; before that 2026-09-10 over #1598, #1596, #1604 and #1603, adopting #1598's post-review names segment_by_derivative / seed_cosine_similarities). Independent of #1599. Split 2026-09-10: eviction is #1617, stacked on this PR, and the second half (context parts, block kinds, segmenter/deriver tables) is #1611, stacked on #1617. Stands in for [vector store 2/13] #1621, closed: this PR routes the filters and writes the record the way the vector store slices from 3 on assume, and those slices rebase onto it.

What changes

Data models (event_memory/data_types.py)

• Event, Segment, Derivative gain session_id: str (required, non-empty) and source_id: str | None, bounded to ID_MAX_BYTES by the model and copied verbatim by the segmenter and the deriver; every field carries a description, positions are validated non-negative, and timestamps are AwareDatetime: a naive value is rejected on the models and on the typed bounds since and until, since it names no instant and a guessed zone would silently shift an event in the order. The session is required because it is the unit the order is partitioned into: every event belongs to one stream. A null source is a missing record key, so every backend's "missing" implements IS NULL. A typed id list (session_ids, source_ids) holds ids only; a list left None admits everything and an empty one nothing. Property values stay None-free, so absence is the only no-value state.
• Derivative loses context and properties: a deriver reads the segment's context while composing the text it embeds, and the memory reads a derivative's timestamp, session, source and block kind for the record; nothing read the copies. A derivative is the derived content and the fields a record carries.
• QueryHit(score, seed, neighborhood) replaces ScoredSegmentContext and QueryResult: the segment the query matched and the Neighborhood(before, after) around it, the shape expand returns, so walking further from a hit composes with expand; window() is the flat list for rendering.

Reserved keys (common/property_keys.py, event_memory.py)

• property_keys.py is taken as-is from edwinyyyu/MemMachine@27b3279b.
• Every system value a search filters on at the vector stage is in the record under a reserved key naming the memory that writes it, em for brevity within the identifier budget: memmachine_em_timestamp, memmachine_em_session, memmachine_em_source, memmachine_em_block_kind. expected_vector_store_collection_schema declares the four, and a record carries the four and nothing else: the user's properties stay on the segment, where property_filter reads them, so the collection schema a memory needs is exactly the four (the server's locator declares no more; the adapter's _episode_uid-style fields are user properties to the memory). _timestamp is gone; a user key in the namespace is rejected at _validate_events (through validate_user_property_key, so an illegal key now fails before any segment is written instead of at the vector upsert).
• system_predicates(...) builds the tree the vector store gets from the typed values. The inverse, recovering typed values from a tree a caller wrote, is not built: nothing calls it.

Segment store

• segment_store_sg gains session_id VARCHAR(255) NOT NULL, source_id VARCHAR(255) NULL, block_kind VARCHAR(255) NOT NULL (the type of partition_key and of the vector stores' key columns, for every string column the store compares on: on PostgreSQL the same storage and index behavior as text plus the length check, on SQLite text affinity either way, and on MySQL, SQL Server and Oracle the only one of the two that can be an index key), projected from the segment at insert (block_kind from segment.block.kind, never a separate input). segment_store_sg__in_se_ts_ev_ix_of (incarnation, session_id, timestamp, event_uuid, index, offset) replaces the timestamp ordering index, since every walk pins a session. No index for a walk filtered by source or by kind: the store indexes the reads it has (decision 7). Every index is one a read chooses (measured below); none on source_id alone or on block_kind.
• The total order is (timestamp, event_uuid, index, offset). A walk is confined to the seed's session.
• Two reads with different jobs: get_segments(uuids, *, since, until, session_ids, source_ids, block_kinds, property_filter) -> dict[UUID, Segment] is a filtered lookup, and get_segment_neighborhoods(seed_uuids, *, before, after, since, until, source_ids, block_kinds, property_filter) -> dict[UUID, Neighborhood] walks the order around seeds named by uuid: a seed is located whether or not it passes any filter, the filters select the neighbors, and an unknown seed is absent. Filters select what a read returns: a search fetches its seeds with the filters and walks from the ones that pass. session_ids is the lookup's visibility filter and is not a walk parameter: a walk pins its seed's session, so a session filter on it could only repeat the confinement or empty the answer. since is inclusive, until exclusive, both bound as UTC instants. An empty source_ids / block_kinds admits nothing.
• Migrations: none. startup() keeps create_all, so the new columns exist on a fresh database and an existing speedkick database is recreated, as design/segment_store_shared_tables.md already says. Schema migration is deferred until the lifecycle/DDL changes.

EventMemory

• FormatOptions is DateTimeFormat: date and time styles, locale and zone, and nothing else, frozen (a value); every parameter that took it is datetime_format, and the derivers default to one shared _DATE_ONLY instance (a full date and no time) rather than taking None. DateTimeFormat and DateTimeStyle spell a date and a time as two words, as CLDR spells dateTime; datetime_format spells them as one, as Python does. The segmenter's unused format parameter, a pass-through since the memory took one, is gone.
• EventMemoryParams: reranker leaves, and so does the per-call format_options on encode_events: the deriver owns the format of what it embeds (WholeTextDeriver(datetime_format) and SentenceTextDeriver(datetime_format), default a full date and no time, the shipped ingest recipe), since a deriver decides the text it embeds and one format per memory would assume every deriver wants the same one. Display rendering keeps taking the caller's options per call.
• encode_events(events): segment, derive, embed, add_segments, upsert. Every call adds; encoding an event a second time stores a second copy, as speedkick's encode does, and a follow-up rejects a reused event uuid at the store instead. The branch's serialize_encode lock is not ported.
• query(query, *, vector_search_limit, min_cosine_similarity, expand_context, since, until, session_ids, source_ids, block_kinds, property_filter) -> list[QueryHit]: the vector stage only. Seeds are resolved through get_segment_uuids_by_derivative_uuids, as on Answer with cosine scores and uuids, not vectors and stale properties (speedkick) #1598; a derivative whose segment is gone is skipped. The seeds are fetched with the filters through get_segments, then walked from with get_segment_neighborhoods; no walk is asked for when expand_context is zero.
• rerank(query, hits, *, reranker, datetime_format): static, the second stage for a caller with a reranker; every hit comes back rescored in descending score, and cutting and thresholding are the caller's.
• expand(seed_uuid, *, before, after, since, until, session_ids, source_ids, block_kinds, property_filter) -> Neighborhood, the filters a search takes: the seed is a segment uuid, walked from after a lookup with session_ids alone when sessions are named, so a seed outside them is not found; LookupError for an unknown seed. Expansion is by segment only; an event is never a seed.
• render_segments(segments, *, datetime_format) replaces string_from_segment_context / string_from_segment_contexts, with _is_continuation deciding headers (a new header when the piece is not the very next one of the same event; no gap marker). build_query_result_context and string_from_query_result are gone.

Server (long_term_memory.py): _episode_to_event sets source_id = producer_id, writes session_id = DEFAULT_SESSION_ID (memmachine_default: the API carries no conversation id, so a partition's events are one stream under the one reserved session name, a stop-gap until the API requires a session; any other name is a caller's to use) and keeps the producer context as before; _search_scored_event lifts the mapped fields out of the filter (decision 3), calls query then EventMemory.rerank when a reranker is configured, and reads QueryHits through seed and window().

Decisions the handoff left open, and deviations

Each of these is my call and can be reversed:

1. No migration. The handoff asked for a revision; per the 2026-09-10 decision, migrations wait for the lifecycle/DDL work and speedkick databases are recreated. Rows written before this change would not decode anyway (their discriminators were context_type / block_type), so an upgrade would have had to rewrite payloads as well as add columns. Repo-wide boot-path DDL policy stays with Schema provisioning runs from every process's boot path: create_all races on cold boot and never evolves a table, Alembic runs destructive migrations on first use #1570.
2. ID_MAX_BYTES = 255 bounds session_id and source_id (the handoff said "the same limit as a property string value", and no such limit exists on speedkick). A value that long fits the store's 255-character key columns, and the database enforces the bound the model does. When properties.max_string_bytes lands as a setting this should follow it.
3. property_filter is the segment store's post-filter and never reaches the vector store, and user properties are never written there. The typed filters name the memory's own fields, which the record carries under reserved keys, so the vector store evaluates them during the search; property_filter is the caller's, over properties the vector store does not hold, applied by the segment store to the seeds and their neighbors. A vector record is the vector and the four reserved keys, so a store that declares what it indexes and refuses the rest ([vector store 9/13]) takes every record the memory writes. A selective filter returns fewer than vector_search_limit hits rather than costing more work. The server's legacy API carries the fields its ingestion maps onto the event in the filter tree, so LongTermMemory lifts every top-level conjunct on a mapped field back into the typed parameter before the call: timestamp and created_at bounds into since and until, producer_id = and IN into source_ids (their intersection when several; empty admits nothing); other operators, and anything under a disjunction or negation, stay post-filters. The store-side field mapping (m.<key> to the properties JSON, bare timestamp to the column, other bare names to _<field>) is kept, since the server still writes its fields that way.
4. The event index stays (incarnation, event_uuid), a lookup index; get_segment_uuids_by_event_uuids promises no order, since nothing reads one.
5. get_segment_neighborhoods with zero counts still locates the seeds and returns two empty lists for each, so presence in the mapping means the seed exists.
6. The PostgreSQL walk keeps speedkick's shape: the seeds subquery selects the seed rows by uuid inside the statement, now with the seed's session_id, and the lateral pins session_id to it, an equality the planner parameterizes per seed so the session-led index serves each; one statement per direction for every seed. An earlier version of this PR ran a statement pair per distinct seed session and cost a search with twenty seeds in twenty sessions 53 ms; a bound unnest row set measured the same as the subquery and was dropped for it. A fresh PostgreSQL table misplans until its first ANALYZE, whatever the indexes: with no statistics the planner tied the primary key with (incarnation, event_uuid) for the link table's foreign-key check and scanned the partition per link (ingest quadratic in the rows imported inside that window: 129 to 1,249 ms per thousand over a 20,000-row import), and ran a sequential scan for the lookup by uuid (search-shaped read 29 ms against 7.9 after ANALYZE). Autovacuum analyzes a table within about a minute of its first rows and the cached plans are replaced (measured on one connection: 403 ms per batch before, 35 after); an initial bulk import should be followed by ANALYZE, PostgreSQL's documented practice. Leading the indexes with another column, or dropping the constraint, only changes the planner's candidates and was not kept.
7. Indexes for the reads the store has: the primary key, lookup by event, and the session-led walk. A source- or kind-pinned walk index would serve a walk filtered by one value of it, which no measured workload runs; the two are treated alike, and either earns its index when a workload shows it (a one-off CREATE INDEX CONCURRENTLY, migration-time).
8. _normalize_column_value from the branch is not ported: on speedkick the filter nodes already normalize datetimes at construction (the FilterExpr contract), and the typed bounds normalize in the store. The test that a zoned bound compares as an instant on SQLite covers the new path.
9. Not ported, per the handoff: get_neighbor_events, AnnotationContext / CompositeContext / find_contexts, the gap marker, the serialize_encode lock. The claude-memory engine's move of session and author into the two fields lives on agentic_expansion, not on this branch.

Verification

• uv run pytest packages/server/server_tests packages/common packages/client: 2229 passed, 2 skipped (the default not integration selection; SQLite).

• ... pytest packages/server/server_tests/memmachine_server/episodic_memory/event_memory/segment_store -m "integration or not integration": 220 passed (SQLite and PostgreSQL via testcontainers).

• ty check --project packages/server, ruff check, ruff format --check: clean.

• Review rounds of 2026-09-11 to 2026-09-14, one commit per iteration on top of the original (field descriptions and model bounds, id lists without None, rerank without limit and min_score, the lookup-and-walk split and QueryHit, the required session, DateTimeFormat, the reserved keys, the post-filter and the record that carries only the reserved keys, the key column type): the suites rerun after each; the counts above are from the last.

• The store was benchmarked against speedkick's (a8322a7) on the same data: 20,000 segments over 40 sessions and 8 sources, ingested in batches of 1,000 with one derivative link each; 20 seeds per search-shaped read with a 2+4 walk, one seed per expand-shaped read with a 5+5 walk, 60 timed calls after 10 warm ones, old and new alternating, two rounds; PostgreSQL on one testcontainer with the table analyzed after ingest, SQLite on a file. Medians per call, the two rounds' range:

  operation	SQLite old	SQLite new	PostgreSQL old	PostgreSQL new
  ingest 1,000 segments + links	28-32 ms	38-40 ms	30-32 ms	36-38 ms
  segment uuids by 20 derivatives	0.5	0.5	1.0	1.0
  segment uuids by 20 events	0.5	0.5	0.9	1.0
  search-shaped, unfiltered	15-15	16-16	5.3-5.7	7.6-9.6
  search-shaped, m.tag = 'a'	6.1-6.4	7.0-7.2	41-45	8-10
  expand-shaped, unfiltered	1.4	1.4-1.5	2.8-3.2	3.0-3.5
  expand-shaped, m.src = 'p3' (JSON)	0.7	1.7	1.2-1.5	3.4-4.1
  expand-shaped, source_ids=['p3'] (typed)		1.5		3.0
  delete 100 segments	12-12	14-15	3.0-3.2	3.5-3.9

  What the differences are: ingest and delete pay for a six-column walk index in place of a five-column one, plus the three new columns (the table's figures were taken with a second, source-pinned walk index that was dropped afterwards, so the ingest and delete differences are an upper bound). A search-shaped read runs one statement more than before, the walk's locate of seeds the lookup already fetched, the price of a walk keyed by uuid; a filtered one is faster because the filter selects the seeds before any walk, where the old store filtered inside every walk. A walk is confined to the seed's session, so with sessions interleaved in time its neighbors are rows 40 apart instead of adjacent ones, which is the JSON-filtered expand's difference (a session-led scan past the other sources versus adjacent rows); the typed source filter takes the source-pinned index and pays only the statements. The unfiltered PostgreSQL search was 53 ms before the one-statement walk and the fresh-table ingest 690 ms per batch before the index order, both in this PR's own history (decision 6).

• A walk filtered by source or by kind scans past the session's other rows on the session-led index; measured on PostgreSQL with 200,000 segments, a source-pinned walk index turned a rare-source walk from 0.106 ms into 0.031 ms and left a common-source walk unchanged, and a walk naming several sources or none used the session-led index either way. A walk index is paid for on every insert and added later in a one-off background build (CREATE INDEX CONCURRENTLY, 2.5 s per million rows here), so none is kept until a workload shows walks filtered by source or by kind.

• Each new store assertion was run against a mutated store (session predicate removed; until made inclusive; neighborhoods seed filtered; bound left unnormalized) and failed there.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MuAu353FiSmCJjLX1LWDQW
https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE