Skip to content

(Depends on #1617) Context parts, block kinds and kind-keyed segmenter and deriver tables (speedkick) - #1611

Closed
edwinyyyu wants to merge 72 commits into
MemMachine:speedkickfrom
edwinyyyu:feat/event-memory-blocks-speedkick
Closed

edwinyyyu wants to merge 72 commits into
MemMachine:speedkickfrom
edwinyyyu:feat/event-memory-blocks-speedkick

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

The second half of design/event_memory_handoff.md (branch design/tenant-lifecycle, PR #1579), split out of #1597 on 2026-09-10. Stacked on #1597 and, on it, #1617 (eviction): their commits come first, and this PR's are the last four: the rebased original, the block-typing fix, the context class, and the table wording on the memory's parameters. Design: design/components/blocks.md ("Processing") and design/components/context.md ("Rendering").

What changes

Blocks (event_memory/data_types.py)

  • Block is an ABC with kind as the discriminator every registered family uses and render(options) -> str | None; the union stays closed over TextBlock. The segment row's block_kind column reads block.kind.
  • Event.blocks and Segment.block are typed Block, the ABC: a before-validator decodes encoded data through the registered union, and SerializeAsAny serializes the concrete kind, so a handler's typed block assigns without narrowing. (RegisteredBlock as the field type left the branch with three type errors in the store and the segmenter, now gone.)

Context parts (event_memory/data_types.py)

  • Context is a class built from parts and read by kind (Context(Author(...)), context.get("author"), context.with_part(part)), never None; a caller never writes a part's key, so cannot write a wrong one, and two parts of one kind are rejected at construction. Author is the first registered kind; an unregistered kind decodes to UnknownPart and round-trips unchanged. The class owns its wire form, {kind: fields}, through a Pydantic core schema, so a model field of the type accepts an instance or the encoded form and serializes to the encoded form. ProducerContext, NullContext and the discriminated union are gone.
  • Server (long_term_memory.py): the producer id is an Author part beside source_id, so rendered segments keep their producer: text shape.

Segmenter and deriver (segmenter/, deriver/)

  • Segmenter and Deriver are tables from block kind to handler, built from handlers in order with a later handler replacing an earlier one for its kind. A kind with no handler passes through as one segment, unchanged, and derives nothing; a step never raises on a kind.
  • BlockSegmenter[B] and BlockDeriver[B] are the handler contracts, one kind each (kind: ClassVar[str]), typed by the block class: split(event, block) -> list[Piece] and derive(segment, block) -> list[str]. The table builds every envelope, so a handler decides only the pieces (Piece(offset, block)) or the texts.
  • TextSegmenter, WholeTextDeriver and SentenceTextDeriver keep their names as text handlers. PassthroughSegmenter is gone, and so is type: passthrough: one segment per block is the identity segmentation, the complete answer for any kind, and a default gets no name. segmenter omitted in configuration means it; segmenter: {type: text, max_chunk_length: N} selects the splitter.

Composition (event_memory.py, formatting.py)

  • A derivative is always text: Derivative.block becomes text, plus block_kind (the segment's) for the vector record. Every derivative gets the same context processing.
  • format_header (formatting.py) is the one composition point for the embedded text and the rendered header: the timestamp, written per a DateTimeFormat, then the context parts parts names in that order (default ("author",)), then the content. Parts carry no order of their own; a part not listed contributes nothing; a new kind is placed by listing it. parts is a parameter of the composers, not a field of DateTimeFormat, which writes a timestamp and nothing else. A BlockDeriver handler owns its datetime_format and parts, so a kind or a handler can embed under its own composition, and the Deriver table composes each derivative's text with them; Derivative.text is the text to embed. render_segments takes the caller's datetime_format and parts per call. EventMemoryParams.format_options is gone.

Decisions

  1. One kind per handler. The table has already dispatched, so a multi-kind handler would only re-narrow and grow back the NotImplementedError branch this removes; kind and block class are one-to-one, so the handler's B gives it a typed block.
  2. Handlers return pieces and texts, not segments and derivatives. Every envelope field is the event's or the segment's by contract; a handler that built them could only get them wrong.
  3. Derivatives are text; composition lives in the deriver table, with the handler's options. The text deriver used to rebuild the header itself, which was the one place a block kind and a part kind met. The table composing through one function removes that matrix: a part's contribution is per part kind, a block's content is per block kind, and there is one composition point. The format is the handler's because the handler decides what is embedded and different kinds or handlers may want different formats.
  4. Order is explicit and owned by the format options, not by the parts; timestamp first and content last are fixed. A handler's parts order what it embeds, a caller's order the display.
  5. Unhandled kinds: identity for the segmenter, nothing for the deriver, never an error at ingest. The deriver's hole is one a kind's registration should close by declaring its deriver or none (blocks.md, proposed); raising at ingest would fail a batch on replayed history whose kind the server no longer registers.
  6. Filter translation lives in event_memory/utils.py (moved from system_filters.py in Add session, source and expansion to EventMemory (speedkick) #1597's history).
  7. Not in this change: the kind table and memmachine.block_kinds entry points, UnknownBlock, base-from-defaults and per-kind tenant options (blocks.md, proposed).

Verification

  • uv run --frozen pytest packages/server/server_tests/memmachine_server/episodic_memory packages/server/server_tests/memmachine_server/common/filter packages/server/server_tests/memmachine_server/common/test_property_keys.py packages/server/server_tests/memmachine_server/common/configuration: 620 passed (SQLite).
  • ... pytest packages/server/server_tests/memmachine_server/episodic_memory/event_memory/segment_store -m integration: 111 passed (PostgreSQL via testcontainers).
  • ty check --project packages/server, ruff check, ruff format --check: clean.
  • Rebased 2026-09-11 onto Add session, source and expansion to EventMemory (speedkick) #1597's review round, then again onto its lookup-and-walk split and QueryHit (seed_index, cosine_similarity_threshold, id lists without None, the unconfined walk from a seed with no session, rerank without limit and min_score); the same suites rerun at the counts above.
  • Known, pre-existing: TextSegmenter's splitter keeps strip_whitespace=True, so pieces rebuild a block only up to boundary whitespace; the test asserts that and the one-line fix changes embedded chunk text, so it is a separate decision.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE

@edwinyyyu
edwinyyyu force-pushed the feat/event-memory-blocks-speedkick branch 14 times, most recently from 212ba7f to e618650 Compare September 11, 2026 17:37
@edwinyyyu edwinyyyu changed the title Context parts, block kinds and kind-keyed segmenter and deriver tables (speedkick) (Depends on #1597) Context parts, block kinds and kind-keyed segmenter and deriver tables (speedkick) Sep 11, 2026
@edwinyyyu
edwinyyyu force-pushed the feat/event-memory-blocks-speedkick branch 12 times, most recently from 5c774da to c6583a8 Compare September 12, 2026 00:58
@edwinyyyu
edwinyyyu marked this pull request as draft September 14, 2026 17:25
@edwinyyyu
edwinyyyu force-pushed the feat/event-memory-blocks-speedkick branch from c6583a8 to bfe69ad Compare September 14, 2026 18:50
@edwinyyyu
edwinyyyu force-pushed the feat/event-memory-blocks-speedkick branch from 9e96a53 to f747e98 Compare September 15, 2026 20:27
@edwinyyyu
edwinyyyu force-pushed the feat/event-memory-blocks-speedkick branch from f747e98 to 620730f Compare September 15, 2026 21:36
@edwinyyyu
edwinyyyu force-pushed the feat/event-memory-blocks-speedkick branch from 620730f to fa616d3 Compare September 15, 2026 21:37
Two `if`s say what a loop over a tuple of names said; `_in_values` is a
static method under `_row_conditions`, the one place that calls it; and
the comment on an empty lookup says what the registry read is for.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
@edwinyyyu
edwinyyyu force-pushed the feat/event-memory-blocks-speedkick branch from fa616d3 to 18770cf Compare September 15, 2026 21:53
The order that put the incarnation second removed a competitor from the
planner's candidates for the link table's foreign-key check; it did not
make the planner choose well, statistics do. A fresh PostgreSQL table
misplans until its first ANALYZE whatever the indexes (the lookup by
uuid runs a sequential scan in that window too), and autovacuum's first
pass, or an ANALYZE after an initial import, ends it for the table's
lifetime. So the indexes read as the rest of the store does, scoped by
the incarnation first, and the PR body states the practice.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
@edwinyyyu
edwinyyyu force-pushed the feat/event-memory-blocks-speedkick branch from 18770cf to b18c2ee Compare September 15, 2026 22:25
edwinyyyu and others added 8 commits September 16, 2026 09:43
The source-pinned walk index served a walk filtered by one source, and
no measured workload runs one; the session-led index serves every walk,
and a filter on source or kind scans past the session's other rows, tens
of microseconds on a 200,000-row table. An index is paid for on every
insert for as long as it exists, and adding one later is a one-off
background build (2.5 s per million rows on PostgreSQL, concurrently),
so the store indexes the reads it has: the primary key, lookup by
event, and the walk. A walk filtered by source or by kind earns its
index when a workload shows it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
`encode_events` no longer forgets the batch's events before encoding
them: every call adds, and encoding an event a second time stores a
second copy, as speedkick's encode does. The forget-first step widened a
race that speedkick already has: a forget that runs between an encode's
segment commit and its vector upsert orphans the encode's records, and
with encode forgetting first, a concurrent re-encode of one event took
that path too. A follow-up moves the upsert inside the write transaction
and rejects a reused event uuid at the store; until then encode is a
plain insert.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
Ported from the agentic_expansion branch, cosine only. EventMemory
takes an optional EvictionOptions: a cosine similarity threshold at or
above which eviction is considered, the number of stored derivatives
at or above it fetched per new derivative, and how many of a new
derivative and those stored ones to keep when there are more. None
keeps every derivative and issues no query.

At encode, after embedding: the batch's predecessors of each
derivative are computed from the batch's own embeddings, earlier
indices only, so a batch evicts what serial ingestion would; one
batched neighbor query fetches the stored derivatives at or above the
threshold; the stored ones' timestamps come from their segments
through the store's lookup, since the vector store answers uuids and
scores only, and a neighbor whose segment is gone is not a member;
then, per derivative, the members over target_size are trimmed from
the temporal middle, the earliest target_size // 2 and the latest
remainder kept. Displaced stored records are deleted from the vector
store and unlinked from their segments with delete_derivatives, which
the store contract, the SQLAlchemy store and the fake gain; skipped
batch derivatives are never written. The batch is sorted by timestamp
first, so the predecessor rule matches serial order.

Tests: the branch's eviction tests on the new shapes, and
delete_derivatives on both dialects, including on a stale handle.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
…s (speedkick)

The second half of design/event_memory_handoff.md, on top of the
session/source/expansion/eviction change: the content model and the
processing steps become registered families keyed by kind.

Blocks. `Block` is an ABC with `kind` as the discriminator every
registered family uses and `render(options) -> str | None`; the union
stays closed over `TextBlock`; the segment row's `block_kind` reads
`block.kind`.

Context. `Context = Mapping[str, ContextPart]`, keyed by part kind,
never None; `Author` is the first registered kind, an unregistered
kind decodes to `UnknownPart` and round-trips unchanged; `with_part` composes one. `ProducerContext`, `NullContext` and the
discriminated union go; the codec encodes a context as `{kind:
fields}`. The server puts the producer id in an `Author` part beside
`source_id`, so rendering keeps its `producer: text` shape.

Segmenter and deriver. `Segmenter` and `Deriver` are tables from block
kind to handler, built from handlers in order with a later handler
replacing an earlier one for its kind. `BlockSegmenter[B]` and `BlockDeriver[B]` are the one-kind
handler contracts, typed by the block class: `split(event, block) ->
list[Piece]` and `derive(segment, block) -> list[str]`. The table
builds every envelope; a handler decides pieces or texts and nothing
else. A kind with no handler passes through as one segment and derives
nothing; nothing raises on a kind. `TextSegmenter`, `WholeTextDeriver`
and `SentenceTextDeriver` keep their names as `text` handlers;
`PassthroughSegmenter` and its configuration name go: one segment per
block is the identity segmentation, and a default gets no name, so an
omitted `segmenter` means it.

Composition. A derivative is text (`Derivative.text`, the text to
embed, plus the segment's `block_kind` for the record), so every
derivative gets the same context processing. `format_header` is the
one composition point for the embedded text and the rendered header:
the timestamp, then the context parts `parts` names in that order
(default `("author",)`), then the content. Parts carry no order; a part
not listed contributes nothing; a new kind is placed by listing it.
`parts` is a parameter of the composers, the handler and `render`,
not a field of `DatetimeFormat`, which writes a timestamp and nothing
else. A handler owns the composition of what it embeds, its
`datetime_format` and `parts`, so a kind or a handler can embed under
its own; the Deriver table composes each derivative's text with them;
rendering for display takes the caller's per call.

Tests: table dispatch, override order, envelope copying; text handlers returning bare
content; composition order, unlisted parts, kind-name validation, and
the anchor over a derivative; context and block round-trips including
an unregistered part kind.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
RegisteredBlock as the field type left three type errors where a
handler's or the codec's Block was assigned to the field. The fields
are typed Block, decoded through the registered union by a
before-validator and serialized as the concrete kind; the handler
tests keep a typed block in hand instead of narrowing the field.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
A context was a mapping the caller keyed by hand, and the rule that
the key equals the part's kind lived in prose. Context is a class now:
built from parts, read by kind, so there is no key to get wrong; two
parts of one kind are rejected at construction. It owns its wire form
through a Pydantic core schema, so a model field of the type accepts
an instance or `{kind: fields}` and serializes to `{kind: fields}`,
and the per-model validators, `with_part`, `encode_context` and
`decode_context` go.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
@edwinyyyu

Copy link
Copy Markdown
Contributor Author

Closed in favor of its main port, #1687, which carries this PR's content at its final tip; the review history stays here. See the "Source and review" section of #1687 for what was reviewed and what was adapted.

@edwinyyyu edwinyyyu left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

packages/server/src/memmachine_server/episodic_memory/event_memory/segmenter/passthrough_segmenter.py survives this change with no consumer: the diff removes PassthroughSegmenterConf, the type: passthrough branch in service_locator.py and test_passthrough_segmenter.py, but not the module, so git grep PassthroughSegmenter at this head hits only its own definition. The description says "PassthroughSegmenter is gone"; the module should go with it. long_term_memory.py (_episode_to_event neighborhood) also still says "under non-passthrough segmenters". Same finding on the port, #1687.

@edwinyyyu

Copy link
Copy Markdown
Contributor Author

The passthrough module is removed on the port, #1687, in 528f250, together with the long-term memory comment that named it. This PR stays closed as reviewed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant