Skip to content

Answer with cosine scores and uuids, not vectors and stale properties (speedkick) - #1598

Merged
edwinyyyu merged 5 commits into
MemMachine:speedkickfrom
edwinyyyu:feat/vector-store-cosine-scores-speedkick
Sep 10, 2026
Merged

edwinyyyu merged 5 commits into
MemMachine:speedkickfrom
edwinyyyu:feat/vector-store-cosine-scores-speedkick

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Replaces #1591 and #1593, resliced. The engine that was in front is now behind, so it never has to ship a method it cannot implement.

Purpose of the change

Three things the vector store was doing that it is not the authority for.

It carried four similarity metrics and shipped one. Every embedder here produces vectors meant to be compared by cosine — OpenAI hard-coded it, Bedrock defaulted to it, SentenceTransformer only reported what the model declared — while every layer that touched a score paid for the other three in direction flags, threshold directions, and per-backend tables mapping the enum onto native metric names.

It handed vectors back. VectorStoreCollection.get had one production caller: the semantic storage read a feature's stored embedding back so it could write the same embedding again with fresh properties, because upsert demands a whole record. return_vector=True was passed there and nowhere else, and VectorSearchEngine.get_vectors existed to serve it.

It handed properties back. Both consumers were reading that copy to get from a search hit to a domain id, and what the copy cost them differed: semantic memory's was of mutable columns, only as fresh as the last write to it, while event memory's was an immutable uuid pairing that stayed correct but duplicated a mapping the segment store already served from an index.

Description

SimilarityMetric is gone. Scores are cosine similarities in [-1, 1], and the names say so: QueryMatch.score → cosine_similarity, query(score_threshold=) → query(min_cosine_similarity=), which no longer needs a direction to be meaningful. The Bedrock embedder's similarity_metric config key goes with it, and the install and configuration docs drop it.

get is removed, with nothing offered in its place, and VectorSearchEngine.get_vectors goes with it. Semantic memory's read-modify-write is gone (see below), so nothing reads a stored vector back at all.

No scoring-by-id entry point takes its place. One would be needed if a caller assembled a candidate set outside the store and asked for those ids to be scored — what a selective-filter plan running above the store would do. Property filtering stays inside the store instead, so the candidate set stays there too, and a store-side regime reaches engine keys directly without a public method addressed by record UUID.

QueryMatch answers a record_uuid and a score. Properties stay stored and filterable — a filter is evaluated against the copy rather than trusted as the record — and are never returned. That ends return_properties, Record on the read path, and set_properties.

Each consumer now owns its own mapping:

  • Event memory used a _segment_uuid property. The segment store already holds that mapping on the derivative's own row, non-null, under a primary key leading on exactly the columns the lookup filters — so the copy bought nothing an indexed read does not. get_segment_uuids_by_derivative_uuids mirrors the forward lookup that was already there.
  • Semantic memory had nothing to reach for: its record uuid was uuid5(namespace, feature_id), one-way, so the feature id rode along in the properties. It now owns a unique vector_uuid column and resolves hits through it. Every property that collection carried was a copy of a column on the feature row, never filtered on, so the payload goes entirely. The delete paths read the column before deleting the rows, since the row is what says which vector record a feature owns.

Two consequences beyond the vector store

The vector graph stores carried a metric per stored embedding, as a companion property beside every vector. That is gone; Node.embeddings holds plain vectors.

NebulaGraph's cosine() cannot take APPROXIMATE and its vector indexes offer only L2 and IP. Cosine similarity between unit vectors is their inner product, so embeddings are normalized on the way in and compared with inner_product() against an IP index: cosine ranking that, unlike cosine(), can be approximate. Normalization happens in _vector_to_gql_literal, the one place any vector becomes a GQL literal, so the stored side and the query side cannot disagree. Its tests skip everywhere, CI included — they need a real NebulaGraph server rather than a testcontainer — so this is the least-exercised part of the change and wants a reviewer who can run against a licensed 5.x server.

Breaking

Data on speedkick does not survive this: existing semantic features have no vector_uuid, and existing collections carry properties nothing reads.

Verification

  • pytest packages/server/server_tests: 1898 passed, 3 skipped, 1 failed.
  • pytest -m integration packages/server/server_tests/memmachine_server/common, against real containers: 282 passed, 106 skipped.
  • ty check --project packages/server: clean.
  • ruff check / ruff format --check: clean.

The one failure is test_get_version. It rejects the .post segment git describe puts into the derived version — here 0.3.9.post2.dev10+g21105d6e6.d20260909 — which the test's regex does not allow. Nothing in this PR touches versioning or tags, and it fails the same way on untouched speedkick.


🤖 Generated with Claude Code

https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 9, 2026
Implements design/event_memory_handoff.md from the tenant-lifecycle
branch: the accepted parts of the server redesign that live in
EventMemory, the segment store and their data models. Nothing is renamed
and nothing outside the event memory is rewired beyond calling the new
API.

Data models. `Event`, `Segment` and `Derivative` carry `session_id` and
`source_id` as first-class nullable fields, copied verbatim down the
pipeline. `Block` is an ABC with `kind` as its discriminator and a
`render`; the union stays closed over `TextBlock`. `Context` is a mapping
from part kind to a registered `ContextPart` (`Author` first; an
unregistered kind decodes to `UnknownPart` and round-trips), replacing
`ProducerContext`, `NullContext` and the discriminated union. `SearchHit`
replaces `ScoredSegmentContext` and `QueryResult`; `Neighborhood` and
`EvictionOptions` are added.

Reserved keys. Every system value a search filters on at the vector
stage sits in the record under a `memmachine_` key (`event_timestamp`,
`event_session`, `event_source`, `block_kind`); the collection schema
declares the four, and a caller key in the namespace is rejected at
`_validate_events`. The derivative's segment and event are not copied
into the record: on this base (MemMachine#1598) the vector store answers uuids and
scores, and the segment store owns those mappings, so seeds are resolved
through `get_segment_uuids_by_derivative_uuids` and eviction reads a
stored neighbor's timestamp from its segment. `system_filters.py` owns
the translation between the typed parameters (`since`, `until`,
`session_ids`, `source_ids`, `block_kinds`) and filter trees on the
reserved keys, in both directions, so either answer to "may a caller
name a system field in a tree" is a small change at an API boundary.

Segment store. `segment_store_sg` gains `session_id`, `source_id` and
`block_kind`, projected from the segment at insert, and the ordering
index becomes `(incarnation, session_id, timestamp, event_uuid, index,
offset)`, the one total order the store exposes. Windows and
neighborhoods are confined to the seed's session, with a null session
one stream. `get_segment_contexts` takes `before`/`after` and the typed
filters; `get_neighbourhoods` returns the neighbors and never the seed,
as two lists; `delete_derivatives` unlinks without touching segments.
Schema changes ship as Alembic revisions under the store, with the
store's own version table, applied by `startup()`; a schema an earlier
release's `create_all` produced is adopted by stamping.

EventMemory. `encode_events` forgets the batch first, so a repeat leaves
one copy, and runs eviction from the agentic_expansion branch, cosine
only: batch predecessors, one bounded neighbor query per derivative, a
cluster over `target_size` trimmed from its temporal middle, displaced
records and their links deleted, skipped ones never written. `query` is
the vector stage and returns hits with the seed's index in its window;
`rerank` is the second stage, static, for a caller with a reranker;
`expand` walks a neighborhood from a segment or event anchor; `render`
replaces the string formatters. The reranker and the per-call format
options leave the constructor and the call, respectively; the deriver's
format is fixed per memory.

Server. `LongTermMemory` builds `Event.source_id` from the producer id
and adds no context; it reranks after `query` and reads hits.

Tests: the branch's neighbor and eviction tests ported to the new shapes
on both dialects, plus session confinement, half-open bounds, instant
comparison of zoned bounds on SQLite, source and kind filters, link
deletion, the migration from the shared-tables layout with the previous
payload shapes, and a model-versus-migrations schema comparison. Each new
store assertion was checked to fail against a mutated store (no session
predicate, an inclusive `until`, a filtered neighborhood seed, an
unnormalized bound).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MuAu353FiSmCJjLX1LWDQW
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-cosine-scores-speedkick branch 5 times, most recently from 0bc0478 to 9bd2594 Compare September 9, 2026 23:20
Two changes to one contract, taken together so its signatures are not
churned twice.

Cosine is now the only similarity. Every embedder MemMachine ships already
produced vectors meant to be compared that way -- OpenAI hard-coded it,
Bedrock defaulted to it, SentenceTransformer only reported what the model
declared -- while every layer that touched a score paid for the other
three metrics in direction flags, threshold directions, and per-backend
tables mapping the enum onto native metric names. `SimilarityMetric` is
gone; scores are cosine similarities in [-1, 1] and the names say so:
`QueryMatch.score` and `SearchMatch.score` become `cosine_similarity`,
`query(score_threshold=)` becomes `query(min_cosine_similarity=)`, which
no longer needs a direction to be meaningful. The Bedrock embedder's
`similarity_metric` config key goes with it, and the install and
configuration docs drop it.

And `VectorStoreCollection.get` is removed, with nothing offered in its
place. It had one production caller: the semantic storage read a feature's
stored embedding back so it could write the same embedding again with
fresh properties, because `upsert` demands a whole record.
`return_vector=True` was passed at that one call site and nowhere else,
and `VectorSearchEngine.get_vectors` existed to serve it. `set_properties`
serves that caller directly -- correcting a record's properties no longer
requires holding its vector -- and on SQLite and sqlite-vec it is an
UPDATE that never touches the index.

No scoring-by-id entry point takes `get`'s place. One would be needed if a
caller assembled a candidate set outside the store and asked for those ids
to be scored, which is what a selective-filter plan running above the
store would do. Property filtering stays inside the store instead, so the
candidate set stays there too, and a store-side regime can reach engine
keys directly without a public method addressed by record UUID.

Two consequences beyond the vector store. The vector graph stores carried
a metric per stored embedding, as a companion property beside every
vector; that is gone and `Node.embeddings` holds plain vectors.
NebulaGraph indexes only L2 and IP and its `cosine()` cannot be
APPROXIMATE, so with cosine alone no index it can build serves a query --
its ANN branch and vector index creation could no longer run and are
removed; search there is always exact KNN.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-cosine-scores-speedkick branch 5 times, most recently from 92fe023 to 52fa5d0 Compare September 10, 2026 00:32
edwinyyyu and others added 4 commits September 9, 2026 17:41
The vector store is not the authority for anything but vectors, so a
consumer that read a record's properties back out of it was reading a copy.
Both consumers did that, and for the same reason: to get from a search hit
back to a domain id. What the copy cost them differed. Semantic memory's
was of mutable columns, correct only as far as the last write to it and
stale from then on. Event memory's was an immutable uuid pairing, so it
stayed correct -- it simply duplicated a mapping the segment store already
served from an index, and spent a reserved property name doing it.

`QueryMatch` therefore answers `record_uuid` and a score. Properties stay
stored and filterable, because a filter is evaluated against the copy
rather than trusted as the record, and are no longer returned. That is the
end of `return_properties`, of `Record` on the read path, and of
`set_properties`, which existed only to keep a copy fresh that nobody
reads now.

Event memory used a `_segment_uuid` property to reach a derivative's
segment. The segment store already holds that mapping on the derivative's
own row, non-null, under a primary key that leads on exactly the columns
the lookup filters -- so the copy bought nothing an indexed read does not,
and `get_segment_uuids_by_derivative_uuids` mirrors the forward lookup that
was already there. The property is gone, and with it a reserved field name.

Semantic memory had no mapping to reach for. Its record uuid was
`uuid5(namespace, feature_id)`, which is one-way, so the feature id had to
ride along in the properties. It now owns a `vector_uuid` column, unique
and minted per feature, and resolves hits through it -- the vector id is
the vector's own, and the caller keeps the correspondence. Every property
that collection carried was a copy of a column on the feature row, never
filtered on, so the payload goes entirely.

The delete paths read that column before deleting the rows, since the row
is what says which vector record a feature owns.

Data on speedkick does not survive this: existing semantic features have no
`vector_uuid`, and existing collections carry properties nothing reads.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL
Cutting `test_similarity_metric_mappings` searched for the next `\ndef ` or
`\nclass ` to find where the function ended. Every test after it is `async
def`, so both searches missed, the end fell through to end-of-file, and the
edit deleted 429 lines: ten test functions, of which four had anything to
do with similarity metrics.

Gone with it were the empty-input cases for adding nodes and edges, the
none-property cases for both, the wrong-collection delete, the
nonexistent-uid read, the multi-property directional search -- and
`test_search_similar_nodes_cosine_metric`, the test for the one metric that
survives this change.

Nothing caught it because these tests skip everywhere, CI included: the
vector support they exercise needs NebulaGraph Enterprise >= 5.0, which the
fixture reaches at NEBULA_HOST and skips without.

Rebuilt from the file on speedkick with the metric stripping applied, then
removing only what has no subject left: the mapping helpers' test, the dot
and manhattan searches, and the ANN search -- Nebula indexes only L2 and
IP, so with cosine alone no index it can build serves a query.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL
Nebula's `cosine()` is KNN-only -- it cannot take APPROXIMATE -- and its
vector indexes offer only L2 and IP. From that I concluded cosine could
never be approximate here and deleted the ANN search branch and vector
index creation as unreachable. That was wrong: cosine similarity between
unit vectors *is* their inner product, so an IP index over normalized
vectors gives cosine ranking with ANN.

Vectors are normalized in `_vector_to_gql_literal`, which is the single
place any vector becomes a literal -- node writes, edge writes, ANN queries
and exact queries all pass through it, so the stored side and the query
side cannot disagree about it. `inner_product()` DESC against an IP index
replaces the metric lookups.

IP rather than L2, though both rank identically over unit vectors
(‖a-b‖² = 2(1-cos), monotone in cos, and verified equal by argsort). The
reason is not numerical: measured over near-duplicate float32 unit vectors,
recovering cosine as 1 - d²/2 is 1.1x worse than reading it off IP, which
is nothing. It is that the contract answers a cosine similarity, and with
IP over unit vectors the index score already is one -- no conversion, no
assumption about whether the engine hands back the distance or its square,
and no result landing outside [-1, 1] needing a clamp.

`test_search_similar_nodes_ann` comes back with it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL
Collapsing the metric enum left an instance attribute that only ever
copied a class constant (self._space = self._SPACE) and, in the Nebula
store, a local that copied a constant into a second local before use.
Read the constants where they are used.

The Nebula metric names move to module scope alongside the package's
other fixed identifier constants; they are neither per-instance nor
overridden.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL
@edwinyyyu
edwinyyyu force-pushed the feat/vector-store-cosine-scores-speedkick branch from 52fa5d0 to 4ff5187 Compare September 10, 2026 00:42
@edwinyyyu
edwinyyyu merged commit abf92a3 into MemMachine:speedkick Sep 10, 2026
32 of 41 checks passed
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 10, 2026
Implements design/event_memory_handoff.md from the tenant-lifecycle
branch: the accepted parts of the server redesign that live in
EventMemory, the segment store and their data models. Nothing is renamed
and nothing outside the event memory is rewired beyond calling the new
API.

Data models. `Event`, `Segment` and `Derivative` carry `session_id` and
`source_id` as first-class nullable fields, copied verbatim down the
pipeline. `Block` is an ABC with `kind` as its discriminator and a
`render`; the union stays closed over `TextBlock`. `Context` is a mapping
from part kind to a registered `ContextPart` (`Author` first; an
unregistered kind decodes to `UnknownPart` and round-trips), replacing
`ProducerContext`, `NullContext` and the discriminated union. `SearchHit`
replaces `ScoredSegmentContext` and `QueryResult`; `Neighborhood` and
`EvictionOptions` are added.

Reserved keys. Every system value a search filters on at the vector
stage sits in the record under a `memmachine_` key (`event_timestamp`,
`event_session`, `event_source`, `block_kind`); the collection schema
declares the four, and a caller key in the namespace is rejected at
`_validate_events`. The derivative's segment and event are not copied
into the record: on this base (MemMachine#1598) the vector store answers uuids and
scores, and the segment store owns those mappings, so seeds are resolved
through `get_segment_uuids_by_derivative_uuids` and eviction reads a
stored neighbor's timestamp from its segment. `system_filters.py` owns
the translation between the typed parameters (`since`, `until`,
`session_ids`, `source_ids`, `block_kinds`) and filter trees on the
reserved keys, in both directions, so either answer to "may a caller
name a system field in a tree" is a small change at an API boundary.

Segment store. `segment_store_sg` gains `session_id`, `source_id` and
`block_kind`, projected from the segment at insert, and the ordering
index becomes `(incarnation, session_id, timestamp, event_uuid, index,
offset)`, the one total order the store exposes. Windows and
neighborhoods are confined to the seed's session, with a null session
one stream. `get_segment_contexts` takes `before`/`after` and the typed
filters; `get_neighbourhoods` returns the neighbors and never the seed,
as two lists; `delete_derivatives` unlinks without touching segments.
No migration: `startup()` keeps `create_all`, and an existing speedkick
database is recreated; schema migration waits for the lifecycle/DDL
changes.

EventMemory. `encode_events` forgets the batch first, so a repeat leaves
one copy, and runs eviction from the agentic_expansion branch, cosine
only: batch predecessors, one bounded neighbor query per derivative, a
cluster over `target_size` trimmed from its temporal middle, displaced
records and their links deleted, skipped ones never written. `query` is
the vector stage and returns hits with the seed's index in its window;
`rerank` is the second stage, static, for a caller with a reranker;
`expand` walks a neighborhood from a segment or event anchor; `render`
replaces the string formatters. The reranker and the per-call format
options leave the constructor and the call, respectively; the deriver's
format is fixed per memory.

Server. `LongTermMemory` builds `Event.source_id` from the producer id
and adds no context; it reranks after `query` and reads hits.

Tests: the branch's neighbor and eviction tests ported to the new shapes
on both dialects, plus session confinement, half-open bounds, instant
comparison of zoned bounds on SQLite, source and kind filters, and link
deletion. Each new store assertion was checked to fail against a mutated store (no session
predicate, an inclusive `until`, a filtered neighborhood seed, an
unnormalized bound).

Rebased 2026-09-10 onto speedkick after MemMachine#1598 merged, adopting its
post-review names (`segment_by_derivative`, `seed_cosine_similarities`).
`common/property_keys.py` and its test, which MemMachine#1598 did not carry into
its merge, are included here.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MuAu353FiSmCJjLX1LWDQW
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 10, 2026
Implements design/event_memory_handoff.md from the tenant-lifecycle
branch: the accepted parts of the server redesign that live in
EventMemory, the segment store and their data models. Nothing is renamed
and nothing outside the event memory is rewired beyond calling the new
API.

Data models. `Event`, `Segment` and `Derivative` carry `session_id` and
`source_id` as first-class nullable fields, copied verbatim down the
pipeline. `Block` is an ABC with `kind` as its discriminator and a
`render`; the union stays closed over `TextBlock`. `Context` is a mapping
from part kind to a registered `ContextPart` (`Author` first; an
unregistered kind decodes to `UnknownPart` and round-trips), replacing
`ProducerContext`, `NullContext` and the discriminated union. `SearchHit`
replaces `ScoredSegmentContext` and `QueryResult`; `Neighborhood` and
`EvictionOptions` are added.

Reserved keys. Every system value a search filters on at the vector
stage sits in the record under a `memmachine_` key (`event_timestamp`,
`event_session`, `event_source`, `block_kind`); the collection schema
declares the four, and a caller key in the namespace is rejected at
`_validate_events`. The derivative's segment and event are not copied
into the record: on this base (MemMachine#1598) the vector store answers uuids and
scores, and the segment store owns those mappings, so seeds are resolved
through `get_segment_uuids_by_derivative_uuids` and eviction reads a
stored neighbor's timestamp from its segment. `utils.py` owns
the translation between the typed parameters (`since`, `until`,
`session_ids`, `source_ids`, `block_kinds`) and filter trees on the
reserved keys, in both directions, so either answer to "may a caller
name a system field in a tree" is a small change at an API boundary.

Segment store. `segment_store_sg` gains `session_id`, `source_id` and
`block_kind`, projected from the segment at insert, and the ordering
index becomes `(incarnation, session_id, timestamp, event_uuid, index,
offset)`, the one total order the store exposes. Windows and
neighborhoods are confined to the seed's session, with a null session
one stream. `get_segment_contexts` takes `before`/`after` and the typed
filters; `get_neighbourhoods` returns the neighbors and never the seed,
as two lists; `delete_derivatives` unlinks without touching segments.
No migration: `startup()` keeps `create_all`, and an existing speedkick
database is recreated; schema migration waits for the lifecycle/DDL
changes.

EventMemory. `encode_events` forgets the batch first, so a repeat leaves
one copy, and runs eviction from the agentic_expansion branch, cosine
only: batch predecessors, one bounded neighbor query per derivative, a
cluster over `target_size` trimmed from its temporal middle, displaced
records and their links deleted, skipped ones never written. `query` is
the vector stage and returns hits with the seed's index in its window;
`rerank` is the second stage, static, for a caller with a reranker;
`expand` walks a neighborhood from a segment or event anchor; `render`
replaces the string formatters. The reranker and the per-call format
options leave the constructor and the call, respectively; the deriver's
format is fixed per memory.

Server. `LongTermMemory` puts the producer id in an `Author` part, as
the producer context was, so rendering keeps its shape; `session_id`
and `source_id` stay unset. It reranks after `query` and reads hits.

Tests: the branch's neighbor and eviction tests ported to the new shapes
on both dialects, plus session confinement, half-open bounds, instant
comparison of zoned bounds on SQLite, source and kind filters, and link
deletion. Each new store assertion was checked to fail against a mutated store (no session
predicate, an inclusive `until`, a filtered neighborhood seed, an
unnormalized bound).

Rebased 2026-09-10 onto speedkick after MemMachine#1598 merged, adopting its
post-review names (`segment_by_derivative`, `seed_cosine_similarities`).
`common/property_keys.py` and its test, which MemMachine#1598 did not carry into
its merge, are included here.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MuAu353FiSmCJjLX1LWDQW
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 10, 2026
Implements design/event_memory_handoff.md from the tenant-lifecycle
branch: the accepted parts of the server redesign that live in
EventMemory, the segment store and their data models. Nothing is renamed
and nothing outside the event memory is rewired beyond calling the new
API.

Data models. `Event`, `Segment` and `Derivative` carry `session_id` and
`source_id` as first-class nullable fields, copied verbatim down the
pipeline. `Block` is an ABC with `kind` as its discriminator and a
`render`; the union stays closed over `TextBlock`. `Context` is a mapping
from part kind to a registered `ContextPart` (`Author` first; an
unregistered kind decodes to `UnknownPart` and round-trips), replacing
`ProducerContext`, `NullContext` and the discriminated union. `SearchHit`
replaces `ScoredSegmentContext` and `QueryResult`; `Neighborhood` and
`EvictionOptions` are added.

Reserved keys. Every system value a search filters on at the vector
stage sits in the record under a `memmachine_` key (`event_timestamp`,
`event_session`, `event_source`, `block_kind`); the collection schema
declares the four, and a caller key in the namespace is rejected at
`_validate_events`. The derivative's segment and event are not copied
into the record: on this base (MemMachine#1598) the vector store answers uuids and
scores, and the segment store owns those mappings, so seeds are resolved
through `get_segment_uuids_by_derivative_uuids` and eviction reads a
stored neighbor's timestamp from its segment. `utils.py` owns
the translation between the typed parameters (`since`, `until`,
`session_ids`, `source_ids`, `block_kinds`) and filter trees on the
reserved keys, in both directions, so either answer to "may a caller
name a system field in a tree" is a small change at an API boundary.

Segment store. `segment_store_sg` gains `session_id`, `source_id` and
`block_kind`, projected from the segment at insert, and the ordering
index becomes `(incarnation, session_id, timestamp, event_uuid, index,
offset)`, the one total order the store exposes. Windows and
neighborhoods are confined to the seed's session, with a null session
one stream. `get_segment_contexts` takes `before`/`after` and the typed
filters; `get_neighbourhoods` returns the neighbors and never the seed,
as two lists; `delete_derivatives` unlinks without touching segments.
No migration: `startup()` keeps `create_all`, and an existing speedkick
database is recreated; schema migration waits for the lifecycle/DDL
changes.

EventMemory. `encode_events` forgets the batch first, so a repeat leaves
one copy, and runs eviction from the agentic_expansion branch, cosine
only: batch predecessors, one bounded neighbor query per derivative, a
cluster over `target_size` trimmed from its temporal middle, displaced
records and their links deleted, skipped ones never written. `query` is
the vector stage and returns hits with the seed's index in its window;
`rerank` is the second stage, static, for a caller with a reranker;
`expand` walks a neighborhood from a segment or event anchor; `render`
replaces the string formatters. The reranker and the per-call format
options leave the constructor and the call, respectively; the deriver's
format is fixed per memory.

Segmenter and deriver. `Segmenter` and `Deriver` become tables from
block kind to handler, built from handlers in order with a later one
replacing an earlier one for its kind; `BlockSegmenter[B]` and
`BlockDeriver[B]` are the one-kind handler contracts, typed by the
block class, returning pieces and blocks while the table builds every
envelope. A kind with no handler passes through as one segment and
derives nothing; nothing raises on a kind. The text handlers keep
their names; `PassthroughSegmenter` goes, the passthrough being the
table's fallback.

Server. `LongTermMemory` makes the producer id the event's `source_id`
and an `Author` part, as the producer context was, so rendering keeps
its shape; `session_id` stays unset. It reranks after `query` and
reads hits.

Tests: the branch's neighbor and eviction tests ported to the new shapes
on both dialects, plus session confinement, half-open bounds, instant
comparison of zoned bounds on SQLite, source and kind filters, and link
deletion. Each new store assertion was checked to fail against a mutated store (no session
predicate, an inclusive `until`, a filtered neighborhood seed, an
unnormalized bound).

Rebased 2026-09-10 onto speedkick after MemMachine#1598 merged, adopting its
post-review names (`segment_by_derivative`, `seed_cosine_similarities`).
`common/property_keys.py` and its test, which MemMachine#1598 did not carry into
its merge, are included here.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MuAu353FiSmCJjLX1LWDQW
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 10, 2026
Implements the session/source/expansion/eviction part of
design/event_memory_handoff.md from the tenant-lifecycle branch. The
context-part and block-kind model and the segmenter/deriver tables
follow in a second change; this one keeps the producer/null context
union and the `block_type` discriminator as they are.

Data models. `Event`, `Segment` and `Derivative` carry `session_id`
and `source_id` as first-class nullable fields, copied verbatim down
the pipeline. `SearchHit(score, seed, segments)` replaces
`ScoredSegmentContext` and `QueryResult`; `Neighborhood(before,
after)` and `EvictionOptions` are added.

Reserved keys. `common/property_keys.py` reserves the `memmachine_`
namespace; every system value a search filters on at the vector stage
sits in the record under a reserved key (`event_timestamp`,
`event_session`, `event_source`, `block_kind`). `utils.py` owns the
translation between the typed filters (`since`, `until`, `session_ids`,
`source_ids`, `block_kinds`) and filter trees; a caller key in the
namespace is rejected before any segment is written.

Segment store. `segment_store_sg` gains `session_id`, `source_id` and
`block_kind` columns, the `(incarnation, session_id, timestamp,
event_uuid, index, offset)` ordering index and a source index; the
total order is `(session_id, timestamp, event_uuid, index, offset)`,
the one order the store exposes, with a null session one stream.
Windows and neighborhoods are confined to the seed's session.
`get_segment_contexts` takes `before`/`after` and the typed filters;
`get_neighbourhoods` returns the neighbors and never the seed, as two
lists; `delete_derivatives` unlinks without touching segments; PG
lateral reads run one statement pair per seed session. No migration:
`startup()` keeps `create_all`, and an existing speedkick database is
recreated; schema migration waits for the lifecycle/DDL changes.

EventMemory. `encode_events` forgets the batch first, so a repeat
leaves one copy, and runs eviction from the agentic_expansion branch,
cosine only: batch predecessors, one bounded neighbor query per
derivative, a cluster over `target_size` trimmed from its temporal
middle, displaced records and their links deleted, skipped ones never
written. `query` is the vector stage and returns hits with the seed's
index in its window; `rerank` is the second stage, static, for a
caller with a reranker; `expand` walks a neighborhood from a segment
or event anchor; `render` replaces the string formatters. The reranker
and the per-call format options leave the constructor and the call,
respectively; the deriver's format is fixed per memory.

Server. `LongTermMemory` sets `Event.source_id` from the producer id
and keeps the producer context; it reranks after `query` and reads
hits.

Tests: the branch's neighbor and eviction tests ported to the new
shapes on both dialects, plus session confinement, half-open bounds,
instant comparison of zoned bounds on SQLite, source and kind filters,
and link deletion. Each new store assertion was checked to fail
against a mutated store (no session predicate, an inclusive `until`, a
filtered neighborhood seed, an unnormalized bound).

Rebased 2026-09-10 onto speedkick after MemMachine#1598 merged, adopting its
post-review names (`segment_by_derivative`, `seed_cosine_similarities`).
`common/property_keys.py` and its test, which MemMachine#1598 did not carry into
its merge, are included here.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MuAu353FiSmCJjLX1LWDQW
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 10, 2026
Implements the session/source/expansion/eviction part of
design/event_memory_handoff.md from the tenant-lifecycle branch. The
context-part and block-kind model and the segmenter/deriver tables
follow in a second change; this one keeps the producer/null context
union and the `block_type` discriminator as they are.

Data models. `Event`, `Segment` and `Derivative` carry `session_id`
and `source_id` as first-class nullable fields, copied verbatim down
the pipeline. `SearchHit(score, seed, segments)` replaces
`ScoredSegmentContext` and `QueryResult`; `Neighborhood(before,
after)` and `EvictionOptions` are added.

Reserved keys. `common/property_keys.py` reserves the `memmachine_`
namespace; every system value a search filters on at the vector stage
sits in the record under a reserved key (`event_timestamp`,
`event_session`, `event_source`, `block_kind`). `utils.py` owns the
translation between the typed filters (`since`, `until`, `session_ids`,
`source_ids`, `block_kinds`) and filter trees; a caller key in the
namespace is rejected before any segment is written.

Segment store. `segment_store_sg` gains `session_id`, `source_id` and
`block_kind` columns, the `(incarnation, session_id, timestamp,
event_uuid, index, offset)` ordering index and a source index; the
total order is `(session_id, timestamp, event_uuid, index, offset)`,
the one order the store exposes, with a null session one stream.
Windows and neighborhoods are confined to the seed's session.
`get_segment_contexts` takes `before`/`after` and the typed filters;
`get_neighbourhoods` returns the neighbors and never the seed, as two
lists; `delete_derivatives` unlinks without touching segments; PG
lateral reads run one statement pair per seed session. No migration:
`startup()` keeps `create_all`, and an existing speedkick database is
recreated; schema migration waits for the lifecycle/DDL changes.

EventMemory. `encode_events` forgets the batch first, so a repeat
leaves one copy, and runs eviction from the agentic_expansion branch,
cosine only: batch predecessors, one bounded neighbor query per
derivative, a cluster over `target_size` trimmed from its temporal
middle, displaced records and their links deleted, skipped ones never
written. `query` is the vector stage and returns hits with the seed's
index in its window; `rerank` is the second stage, static, for a
caller with a reranker; `expand` walks a neighborhood from a segment
or event anchor; `render` replaces the string formatters. The reranker
and the per-call format options leave the constructor and the call,
respectively; the deriver's format is fixed per memory.

Server. `LongTermMemory` sets `Event.source_id` from the producer id
and keeps the producer context; it reranks after `query` and reads
hits.

Tests: the branch's neighbor and eviction tests ported to the new
shapes on both dialects, plus session confinement, half-open bounds,
instant comparison of zoned bounds on SQLite, source and kind filters,
and link deletion. Each new store assertion was checked to fail
against a mutated store (no session predicate, an inclusive `until`, a
filtered neighborhood seed, an unnormalized bound).

Rebased 2026-09-10 onto speedkick after MemMachine#1598 merged, adopting its
post-review names (`segment_by_derivative`, `seed_cosine_similarities`).
`common/property_keys.py` and its test, which MemMachine#1598 did not carry into
its merge, are included here.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MuAu353FiSmCJjLX1LWDQW
Claude-Session: https://claude.ai/code/session_01YBbQgZiCqeoLu83EkbEFHE
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 17, 2026
… (speedkick) (MemMachine#1598)

* Answer with cosine scores, not with stored vectors

Two changes to one contract, taken together so its signatures are not
churned twice.

Cosine is now the only similarity. Every embedder MemMachine ships already
produced vectors meant to be compared that way -- OpenAI hard-coded it,
Bedrock defaulted to it, SentenceTransformer only reported what the model
declared -- while every layer that touched a score paid for the other
three metrics in direction flags, threshold directions, and per-backend
tables mapping the enum onto native metric names. `SimilarityMetric` is
gone; scores are cosine similarities in [-1, 1] and the names say so:
`QueryMatch.score` and `SearchMatch.score` become `cosine_similarity`,
`query(score_threshold=)` becomes `query(min_cosine_similarity=)`, which
no longer needs a direction to be meaningful. The Bedrock embedder's
`similarity_metric` config key goes with it, and the install and
configuration docs drop it.

And `VectorStoreCollection.get` is removed, with nothing offered in its
place. It had one production caller: the semantic storage read a feature's
stored embedding back so it could write the same embedding again with
fresh properties, because `upsert` demands a whole record.
`return_vector=True` was passed at that one call site and nowhere else,
and `VectorSearchEngine.get_vectors` existed to serve it. `set_properties`
serves that caller directly -- correcting a record's properties no longer
requires holding its vector -- and on SQLite and sqlite-vec it is an
UPDATE that never touches the index.

No scoring-by-id entry point takes `get`'s place. One would be needed if a
caller assembled a candidate set outside the store and asked for those ids
to be scored, which is what a selective-filter plan running above the
store would do. Property filtering stays inside the store instead, so the
candidate set stays there too, and a store-side regime can reach engine
keys directly without a public method addressed by record UUID.

Two consequences beyond the vector store. The vector graph stores carried
a metric per stored embedding, as a companion property beside every
vector; that is gone and `Node.embeddings` holds plain vectors.
NebulaGraph indexes only L2 and IP and its `cosine()` cannot be
APPROXIMATE, so with cosine alone no index it can build serves a query --
its ANN branch and vector index creation could no longer run and are
removed; search there is always exact KNN.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Let each store own its own mapping, and stop copying it

The vector store is not the authority for anything but vectors, so a
consumer that read a record's properties back out of it was reading a copy.
Both consumers did that, and for the same reason: to get from a search hit
back to a domain id. What the copy cost them differed. Semantic memory's
was of mutable columns, correct only as far as the last write to it and
stale from then on. Event memory's was an immutable uuid pairing, so it
stayed correct -- it simply duplicated a mapping the segment store already
served from an index, and spent a reserved property name doing it.

`QueryMatch` therefore answers `record_uuid` and a score. Properties stay
stored and filterable, because a filter is evaluated against the copy
rather than trusted as the record, and are no longer returned. That is the
end of `return_properties`, of `Record` on the read path, and of
`set_properties`, which existed only to keep a copy fresh that nobody
reads now.

Event memory used a `_segment_uuid` property to reach a derivative's
segment. The segment store already holds that mapping on the derivative's
own row, non-null, under a primary key that leads on exactly the columns
the lookup filters -- so the copy bought nothing an indexed read does not,
and `get_segment_uuids_by_derivative_uuids` mirrors the forward lookup that
was already there. The property is gone, and with it a reserved field name.

Semantic memory had no mapping to reach for. Its record uuid was
`uuid5(namespace, feature_id)`, which is one-way, so the feature id had to
ride along in the properties. It now owns a `vector_uuid` column, unique
and minted per feature, and resolves hits through it -- the vector id is
the vector's own, and the caller keeps the correspondence. Every property
that collection carried was a copy of a column on the feature row, never
filtered on, so the payload goes entirely.

The delete paths read that column before deleting the rows, since the row
is what says which vector record a feature owns.

Data on speedkick does not survive this: existing semantic features have no
`vector_uuid`, and existing collections carry properties nothing reads.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Restore the Nebula tests a bad edit took with it

Cutting `test_similarity_metric_mappings` searched for the next `\ndef ` or
`\nclass ` to find where the function ended. Every test after it is `async
def`, so both searches missed, the end fell through to end-of-file, and the
edit deleted 429 lines: ten test functions, of which four had anything to
do with similarity metrics.

Gone with it were the empty-input cases for adding nodes and edges, the
none-property cases for both, the wrong-collection delete, the
nonexistent-uid read, the multi-property directional search -- and
`test_search_similar_nodes_cosine_metric`, the test for the one metric that
survives this change.

Nothing caught it because these tests skip everywhere, CI included: the
vector support they exercise needs NebulaGraph Enterprise >= 5.0, which the
fixture reaches at NEBULA_HOST and skips without.

Rebuilt from the file on speedkick with the metric stripping applied, then
removing only what has no subject left: the mapping helpers' test, the dot
and manhattan searches, and the ANN search -- Nebula indexes only L2 and
IP, so with cosine alone no index it can build serves a query.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Keep NebulaGraph's ANN by spelling cosine as an inner product

Nebula's `cosine()` is KNN-only -- it cannot take APPROXIMATE -- and its
vector indexes offer only L2 and IP. From that I concluded cosine could
never be approximate here and deleted the ANN search branch and vector
index creation as unreachable. That was wrong: cosine similarity between
unit vectors *is* their inner product, so an IP index over normalized
vectors gives cosine ranking with ANN.

Vectors are normalized in `_vector_to_gql_literal`, which is the single
place any vector becomes a literal -- node writes, edge writes, ANN queries
and exact queries all pass through it, so the stored side and the query
side cannot disagree about it. `inner_product()` DESC against an IP index
replaces the metric lookups.

IP rather than L2, though both rank identically over unit vectors
(‖a-b‖² = 2(1-cos), monotone in cos, and verified equal by argsort). The
reason is not numerical: measured over near-duplicate float32 unit vectors,
recovering cosine as 1 - d²/2 is 1.1x worse than reading it off IP, which
is nothing. It is that the contract answers a cosine similarity, and with
IP over unit vectors the index score already is one -- no conversion, no
assumption about whether the engine hands back the distance or its square,
and no result landing outside [-1, 1] needing a clamp.

`test_search_similar_nodes_ann` comes back with it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Drop the constant aliases left over from configurable metrics

Collapsing the metric enum left an instance attribute that only ever
copied a class constant (self._space = self._SPACE) and, in the Nebula
store, a local that copied a constant into a second local before use.
Read the constants where they are used.

The Nebula metric names move to module scope alongside the package's
other fixed identifier constants; they are neither per-instance nor
overridden.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 18, 2026
… (speedkick) (MemMachine#1598)

* Answer with cosine scores, not with stored vectors

Two changes to one contract, taken together so its signatures are not
churned twice.

Cosine is now the only similarity. Every embedder MemMachine ships already
produced vectors meant to be compared that way -- OpenAI hard-coded it,
Bedrock defaulted to it, SentenceTransformer only reported what the model
declared -- while every layer that touched a score paid for the other
three metrics in direction flags, threshold directions, and per-backend
tables mapping the enum onto native metric names. `SimilarityMetric` is
gone; scores are cosine similarities in [-1, 1] and the names say so:
`QueryMatch.score` and `SearchMatch.score` become `cosine_similarity`,
`query(score_threshold=)` becomes `query(min_cosine_similarity=)`, which
no longer needs a direction to be meaningful. The Bedrock embedder's
`similarity_metric` config key goes with it, and the install and
configuration docs drop it.

And `VectorStoreCollection.get` is removed, with nothing offered in its
place. It had one production caller: the semantic storage read a feature's
stored embedding back so it could write the same embedding again with
fresh properties, because `upsert` demands a whole record.
`return_vector=True` was passed at that one call site and nowhere else,
and `VectorSearchEngine.get_vectors` existed to serve it. `set_properties`
serves that caller directly -- correcting a record's properties no longer
requires holding its vector -- and on SQLite and sqlite-vec it is an
UPDATE that never touches the index.

No scoring-by-id entry point takes `get`'s place. One would be needed if a
caller assembled a candidate set outside the store and asked for those ids
to be scored, which is what a selective-filter plan running above the
store would do. Property filtering stays inside the store instead, so the
candidate set stays there too, and a store-side regime can reach engine
keys directly without a public method addressed by record UUID.

Two consequences beyond the vector store. The vector graph stores carried
a metric per stored embedding, as a companion property beside every
vector; that is gone and `Node.embeddings` holds plain vectors.
NebulaGraph indexes only L2 and IP and its `cosine()` cannot be
APPROXIMATE, so with cosine alone no index it can build serves a query --
its ANN branch and vector index creation could no longer run and are
removed; search there is always exact KNN.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Let each store own its own mapping, and stop copying it

The vector store is not the authority for anything but vectors, so a
consumer that read a record's properties back out of it was reading a copy.
Both consumers did that, and for the same reason: to get from a search hit
back to a domain id. What the copy cost them differed. Semantic memory's
was of mutable columns, correct only as far as the last write to it and
stale from then on. Event memory's was an immutable uuid pairing, so it
stayed correct -- it simply duplicated a mapping the segment store already
served from an index, and spent a reserved property name doing it.

`QueryMatch` therefore answers `record_uuid` and a score. Properties stay
stored and filterable, because a filter is evaluated against the copy
rather than trusted as the record, and are no longer returned. That is the
end of `return_properties`, of `Record` on the read path, and of
`set_properties`, which existed only to keep a copy fresh that nobody
reads now.

Event memory used a `_segment_uuid` property to reach a derivative's
segment. The segment store already holds that mapping on the derivative's
own row, non-null, under a primary key that leads on exactly the columns
the lookup filters -- so the copy bought nothing an indexed read does not,
and `get_segment_uuids_by_derivative_uuids` mirrors the forward lookup that
was already there. The property is gone, and with it a reserved field name.

Semantic memory had no mapping to reach for. Its record uuid was
`uuid5(namespace, feature_id)`, which is one-way, so the feature id had to
ride along in the properties. It now owns a `vector_uuid` column, unique
and minted per feature, and resolves hits through it -- the vector id is
the vector's own, and the caller keeps the correspondence. Every property
that collection carried was a copy of a column on the feature row, never
filtered on, so the payload goes entirely.

The delete paths read that column before deleting the rows, since the row
is what says which vector record a feature owns.

Data on speedkick does not survive this: existing semantic features have no
`vector_uuid`, and existing collections carry properties nothing reads.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Restore the Nebula tests a bad edit took with it

Cutting `test_similarity_metric_mappings` searched for the next `\ndef ` or
`\nclass ` to find where the function ended. Every test after it is `async
def`, so both searches missed, the end fell through to end-of-file, and the
edit deleted 429 lines: ten test functions, of which four had anything to
do with similarity metrics.

Gone with it were the empty-input cases for adding nodes and edges, the
none-property cases for both, the wrong-collection delete, the
nonexistent-uid read, the multi-property directional search -- and
`test_search_similar_nodes_cosine_metric`, the test for the one metric that
survives this change.

Nothing caught it because these tests skip everywhere, CI included: the
vector support they exercise needs NebulaGraph Enterprise >= 5.0, which the
fixture reaches at NEBULA_HOST and skips without.

Rebuilt from the file on speedkick with the metric stripping applied, then
removing only what has no subject left: the mapping helpers' test, the dot
and manhattan searches, and the ANN search -- Nebula indexes only L2 and
IP, so with cosine alone no index it can build serves a query.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Keep NebulaGraph's ANN by spelling cosine as an inner product

Nebula's `cosine()` is KNN-only -- it cannot take APPROXIMATE -- and its
vector indexes offer only L2 and IP. From that I concluded cosine could
never be approximate here and deleted the ANN search branch and vector
index creation as unreachable. That was wrong: cosine similarity between
unit vectors *is* their inner product, so an IP index over normalized
vectors gives cosine ranking with ANN.

Vectors are normalized in `_vector_to_gql_literal`, which is the single
place any vector becomes a literal -- node writes, edge writes, ANN queries
and exact queries all pass through it, so the stored side and the query
side cannot disagree about it. `inner_product()` DESC against an IP index
replaces the metric lookups.

IP rather than L2, though both rank identically over unit vectors
(‖a-b‖² = 2(1-cos), monotone in cos, and verified equal by argsort). The
reason is not numerical: measured over near-duplicate float32 unit vectors,
recovering cosine as 1 - d²/2 is 1.1x worse than reading it off IP, which
is nothing. It is that the contract answers a cosine similarity, and with
IP over unit vectors the index score already is one -- no conversion, no
assumption about whether the engine hands back the distance or its square,
and no result landing outside [-1, 1] needing a clamp.

`test_search_similar_nodes_ann` comes back with it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Drop the constant aliases left over from configurable metrics

Collapsing the metric enum left an instance attribute that only ever
copied a class constant (self._space = self._SPACE) and, in the Nebula
store, a local that copied a constant into a second local before use.
Read the constants where they are used.

The Nebula metric names move to module scope alongside the package's
other fixed identifier constants; they are neither per-instance nor
overridden.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit abf92a3)
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 18, 2026
… (speedkick) (MemMachine#1598)

* Answer with cosine scores, not with stored vectors

Two changes to one contract, taken together so its signatures are not
churned twice.

Cosine is now the only similarity. Every embedder MemMachine ships already
produced vectors meant to be compared that way -- OpenAI hard-coded it,
Bedrock defaulted to it, SentenceTransformer only reported what the model
declared -- while every layer that touched a score paid for the other
three metrics in direction flags, threshold directions, and per-backend
tables mapping the enum onto native metric names. `SimilarityMetric` is
gone; scores are cosine similarities in [-1, 1] and the names say so:
`QueryMatch.score` and `SearchMatch.score` become `cosine_similarity`,
`query(score_threshold=)` becomes `query(min_cosine_similarity=)`, which
no longer needs a direction to be meaningful. The Bedrock embedder's
`similarity_metric` config key goes with it, and the install and
configuration docs drop it.

And `VectorStoreCollection.get` is removed, with nothing offered in its
place. It had one production caller: the semantic storage read a feature's
stored embedding back so it could write the same embedding again with
fresh properties, because `upsert` demands a whole record.
`return_vector=True` was passed at that one call site and nowhere else,
and `VectorSearchEngine.get_vectors` existed to serve it. `set_properties`
serves that caller directly -- correcting a record's properties no longer
requires holding its vector -- and on SQLite and sqlite-vec it is an
UPDATE that never touches the index.

No scoring-by-id entry point takes `get`'s place. One would be needed if a
caller assembled a candidate set outside the store and asked for those ids
to be scored, which is what a selective-filter plan running above the
store would do. Property filtering stays inside the store instead, so the
candidate set stays there too, and a store-side regime can reach engine
keys directly without a public method addressed by record UUID.

Two consequences beyond the vector store. The vector graph stores carried
a metric per stored embedding, as a companion property beside every
vector; that is gone and `Node.embeddings` holds plain vectors.
NebulaGraph indexes only L2 and IP and its `cosine()` cannot be
APPROXIMATE, so with cosine alone no index it can build serves a query --
its ANN branch and vector index creation could no longer run and are
removed; search there is always exact KNN.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Let each store own its own mapping, and stop copying it

The vector store is not the authority for anything but vectors, so a
consumer that read a record's properties back out of it was reading a copy.
Both consumers did that, and for the same reason: to get from a search hit
back to a domain id. What the copy cost them differed. Semantic memory's
was of mutable columns, correct only as far as the last write to it and
stale from then on. Event memory's was an immutable uuid pairing, so it
stayed correct -- it simply duplicated a mapping the segment store already
served from an index, and spent a reserved property name doing it.

`QueryMatch` therefore answers `record_uuid` and a score. Properties stay
stored and filterable, because a filter is evaluated against the copy
rather than trusted as the record, and are no longer returned. That is the
end of `return_properties`, of `Record` on the read path, and of
`set_properties`, which existed only to keep a copy fresh that nobody
reads now.

Event memory used a `_segment_uuid` property to reach a derivative's
segment. The segment store already holds that mapping on the derivative's
own row, non-null, under a primary key that leads on exactly the columns
the lookup filters -- so the copy bought nothing an indexed read does not,
and `get_segment_uuids_by_derivative_uuids` mirrors the forward lookup that
was already there. The property is gone, and with it a reserved field name.

Semantic memory had no mapping to reach for. Its record uuid was
`uuid5(namespace, feature_id)`, which is one-way, so the feature id had to
ride along in the properties. It now owns a `vector_uuid` column, unique
and minted per feature, and resolves hits through it -- the vector id is
the vector's own, and the caller keeps the correspondence. Every property
that collection carried was a copy of a column on the feature row, never
filtered on, so the payload goes entirely.

The delete paths read that column before deleting the rows, since the row
is what says which vector record a feature owns.

Data on speedkick does not survive this: existing semantic features have no
`vector_uuid`, and existing collections carry properties nothing reads.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Restore the Nebula tests a bad edit took with it

Cutting `test_similarity_metric_mappings` searched for the next `\ndef ` or
`\nclass ` to find where the function ended. Every test after it is `async
def`, so both searches missed, the end fell through to end-of-file, and the
edit deleted 429 lines: ten test functions, of which four had anything to
do with similarity metrics.

Gone with it were the empty-input cases for adding nodes and edges, the
none-property cases for both, the wrong-collection delete, the
nonexistent-uid read, the multi-property directional search -- and
`test_search_similar_nodes_cosine_metric`, the test for the one metric that
survives this change.

Nothing caught it because these tests skip everywhere, CI included: the
vector support they exercise needs NebulaGraph Enterprise >= 5.0, which the
fixture reaches at NEBULA_HOST and skips without.

Rebuilt from the file on speedkick with the metric stripping applied, then
removing only what has no subject left: the mapping helpers' test, the dot
and manhattan searches, and the ANN search -- Nebula indexes only L2 and
IP, so with cosine alone no index it can build serves a query.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Keep NebulaGraph's ANN by spelling cosine as an inner product

Nebula's `cosine()` is KNN-only -- it cannot take APPROXIMATE -- and its
vector indexes offer only L2 and IP. From that I concluded cosine could
never be approximate here and deleted the ANN search branch and vector
index creation as unreachable. That was wrong: cosine similarity between
unit vectors *is* their inner product, so an IP index over normalized
vectors gives cosine ranking with ANN.

Vectors are normalized in `_vector_to_gql_literal`, which is the single
place any vector becomes a literal -- node writes, edge writes, ANN queries
and exact queries all pass through it, so the stored side and the query
side cannot disagree about it. `inner_product()` DESC against an IP index
replaces the metric lookups.

IP rather than L2, though both rank identically over unit vectors
(‖a-b‖² = 2(1-cos), monotone in cos, and verified equal by argsort). The
reason is not numerical: measured over near-duplicate float32 unit vectors,
recovering cosine as 1 - d²/2 is 1.1x worse than reading it off IP, which
is nothing. It is that the contract answers a cosine similarity, and with
IP over unit vectors the index score already is one -- no conversion, no
assumption about whether the engine hands back the distance or its square,
and no result landing outside [-1, 1] needing a clamp.

`test_search_similar_nodes_ann` comes back with it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Drop the constant aliases left over from configurable metrics

Collapsing the metric enum left an instance attribute that only ever
copied a class constant (self._space = self._SPACE) and, in the Nebula
store, a local that copied a constant into a second local before use.
Read the constants where they are used.

The Nebula metric names move to module scope alongside the package's
other fixed identifier constants; they are neither per-instance nor
overridden.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit abf92a3)
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 21, 2026
… (speedkick) (MemMachine#1598)

* Answer with cosine scores, not with stored vectors

Two changes to one contract, taken together so its signatures are not
churned twice.

Cosine is now the only similarity. Every embedder MemMachine ships already
produced vectors meant to be compared that way -- OpenAI hard-coded it,
Bedrock defaulted to it, SentenceTransformer only reported what the model
declared -- while every layer that touched a score paid for the other
three metrics in direction flags, threshold directions, and per-backend
tables mapping the enum onto native metric names. `SimilarityMetric` is
gone; scores are cosine similarities in [-1, 1] and the names say so:
`QueryMatch.score` and `SearchMatch.score` become `cosine_similarity`,
`query(score_threshold=)` becomes `query(min_cosine_similarity=)`, which
no longer needs a direction to be meaningful. The Bedrock embedder's
`similarity_metric` config key goes with it, and the install and
configuration docs drop it.

And `VectorStoreCollection.get` is removed, with nothing offered in its
place. It had one production caller: the semantic storage read a feature's
stored embedding back so it could write the same embedding again with
fresh properties, because `upsert` demands a whole record.
`return_vector=True` was passed at that one call site and nowhere else,
and `VectorSearchEngine.get_vectors` existed to serve it. `set_properties`
serves that caller directly -- correcting a record's properties no longer
requires holding its vector -- and on SQLite and sqlite-vec it is an
UPDATE that never touches the index.

No scoring-by-id entry point takes `get`'s place. One would be needed if a
caller assembled a candidate set outside the store and asked for those ids
to be scored, which is what a selective-filter plan running above the
store would do. Property filtering stays inside the store instead, so the
candidate set stays there too, and a store-side regime can reach engine
keys directly without a public method addressed by record UUID.

Two consequences beyond the vector store. The vector graph stores carried
a metric per stored embedding, as a companion property beside every
vector; that is gone and `Node.embeddings` holds plain vectors.
NebulaGraph indexes only L2 and IP and its `cosine()` cannot be
APPROXIMATE, so with cosine alone no index it can build serves a query --
its ANN branch and vector index creation could no longer run and are
removed; search there is always exact KNN.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Let each store own its own mapping, and stop copying it

The vector store is not the authority for anything but vectors, so a
consumer that read a record's properties back out of it was reading a copy.
Both consumers did that, and for the same reason: to get from a search hit
back to a domain id. What the copy cost them differed. Semantic memory's
was of mutable columns, correct only as far as the last write to it and
stale from then on. Event memory's was an immutable uuid pairing, so it
stayed correct -- it simply duplicated a mapping the segment store already
served from an index, and spent a reserved property name doing it.

`QueryMatch` therefore answers `record_uuid` and a score. Properties stay
stored and filterable, because a filter is evaluated against the copy
rather than trusted as the record, and are no longer returned. That is the
end of `return_properties`, of `Record` on the read path, and of
`set_properties`, which existed only to keep a copy fresh that nobody
reads now.

Event memory used a `_segment_uuid` property to reach a derivative's
segment. The segment store already holds that mapping on the derivative's
own row, non-null, under a primary key that leads on exactly the columns
the lookup filters -- so the copy bought nothing an indexed read does not,
and `get_segment_uuids_by_derivative_uuids` mirrors the forward lookup that
was already there. The property is gone, and with it a reserved field name.

Semantic memory had no mapping to reach for. Its record uuid was
`uuid5(namespace, feature_id)`, which is one-way, so the feature id had to
ride along in the properties. It now owns a `vector_uuid` column, unique
and minted per feature, and resolves hits through it -- the vector id is
the vector's own, and the caller keeps the correspondence. Every property
that collection carried was a copy of a column on the feature row, never
filtered on, so the payload goes entirely.

The delete paths read that column before deleting the rows, since the row
is what says which vector record a feature owns.

Data on speedkick does not survive this: existing semantic features have no
`vector_uuid`, and existing collections carry properties nothing reads.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Restore the Nebula tests a bad edit took with it

Cutting `test_similarity_metric_mappings` searched for the next `\ndef ` or
`\nclass ` to find where the function ended. Every test after it is `async
def`, so both searches missed, the end fell through to end-of-file, and the
edit deleted 429 lines: ten test functions, of which four had anything to
do with similarity metrics.

Gone with it were the empty-input cases for adding nodes and edges, the
none-property cases for both, the wrong-collection delete, the
nonexistent-uid read, the multi-property directional search -- and
`test_search_similar_nodes_cosine_metric`, the test for the one metric that
survives this change.

Nothing caught it because these tests skip everywhere, CI included: the
vector support they exercise needs NebulaGraph Enterprise >= 5.0, which the
fixture reaches at NEBULA_HOST and skips without.

Rebuilt from the file on speedkick with the metric stripping applied, then
removing only what has no subject left: the mapping helpers' test, the dot
and manhattan searches, and the ANN search -- Nebula indexes only L2 and
IP, so with cosine alone no index it can build serves a query.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Keep NebulaGraph's ANN by spelling cosine as an inner product

Nebula's `cosine()` is KNN-only -- it cannot take APPROXIMATE -- and its
vector indexes offer only L2 and IP. From that I concluded cosine could
never be approximate here and deleted the ANN search branch and vector
index creation as unreachable. That was wrong: cosine similarity between
unit vectors *is* their inner product, so an IP index over normalized
vectors gives cosine ranking with ANN.

Vectors are normalized in `_vector_to_gql_literal`, which is the single
place any vector becomes a literal -- node writes, edge writes, ANN queries
and exact queries all pass through it, so the stored side and the query
side cannot disagree about it. `inner_product()` DESC against an IP index
replaces the metric lookups.

IP rather than L2, though both rank identically over unit vectors
(‖a-b‖² = 2(1-cos), monotone in cos, and verified equal by argsort). The
reason is not numerical: measured over near-duplicate float32 unit vectors,
recovering cosine as 1 - d²/2 is 1.1x worse than reading it off IP, which
is nothing. It is that the contract answers a cosine similarity, and with
IP over unit vectors the index score already is one -- no conversion, no
assumption about whether the engine hands back the distance or its square,
and no result landing outside [-1, 1] needing a clamp.

`test_search_similar_nodes_ann` comes back with it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Drop the constant aliases left over from configurable metrics

Collapsing the metric enum left an instance attribute that only ever
copied a class constant (self._space = self._SPACE) and, in the Nebula
store, a local that copied a constant into a second local before use.
Read the constants where they are used.

The Nebula metric names move to module scope alongside the package's
other fixed identifier constants; they are neither per-instance nor
overridden.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit abf92a3)
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 21, 2026
… (speedkick) (MemMachine#1598)

* Answer with cosine scores, not with stored vectors

Two changes to one contract, taken together so its signatures are not
churned twice.

Cosine is now the only similarity. Every embedder MemMachine ships already
produced vectors meant to be compared that way -- OpenAI hard-coded it,
Bedrock defaulted to it, SentenceTransformer only reported what the model
declared -- while every layer that touched a score paid for the other
three metrics in direction flags, threshold directions, and per-backend
tables mapping the enum onto native metric names. `SimilarityMetric` is
gone; scores are cosine similarities in [-1, 1] and the names say so:
`QueryMatch.score` and `SearchMatch.score` become `cosine_similarity`,
`query(score_threshold=)` becomes `query(min_cosine_similarity=)`, which
no longer needs a direction to be meaningful. The Bedrock embedder's
`similarity_metric` config key goes with it, and the install and
configuration docs drop it.

And `VectorStoreCollection.get` is removed, with nothing offered in its
place. It had one production caller: the semantic storage read a feature's
stored embedding back so it could write the same embedding again with
fresh properties, because `upsert` demands a whole record.
`return_vector=True` was passed at that one call site and nowhere else,
and `VectorSearchEngine.get_vectors` existed to serve it. `set_properties`
serves that caller directly -- correcting a record's properties no longer
requires holding its vector -- and on SQLite and sqlite-vec it is an
UPDATE that never touches the index.

No scoring-by-id entry point takes `get`'s place. One would be needed if a
caller assembled a candidate set outside the store and asked for those ids
to be scored, which is what a selective-filter plan running above the
store would do. Property filtering stays inside the store instead, so the
candidate set stays there too, and a store-side regime can reach engine
keys directly without a public method addressed by record UUID.

Two consequences beyond the vector store. The vector graph stores carried
a metric per stored embedding, as a companion property beside every
vector; that is gone and `Node.embeddings` holds plain vectors.
NebulaGraph indexes only L2 and IP and its `cosine()` cannot be
APPROXIMATE, so with cosine alone no index it can build serves a query --
its ANN branch and vector index creation could no longer run and are
removed; search there is always exact KNN.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Let each store own its own mapping, and stop copying it

The vector store is not the authority for anything but vectors, so a
consumer that read a record's properties back out of it was reading a copy.
Both consumers did that, and for the same reason: to get from a search hit
back to a domain id. What the copy cost them differed. Semantic memory's
was of mutable columns, correct only as far as the last write to it and
stale from then on. Event memory's was an immutable uuid pairing, so it
stayed correct -- it simply duplicated a mapping the segment store already
served from an index, and spent a reserved property name doing it.

`QueryMatch` therefore answers `record_uuid` and a score. Properties stay
stored and filterable, because a filter is evaluated against the copy
rather than trusted as the record, and are no longer returned. That is the
end of `return_properties`, of `Record` on the read path, and of
`set_properties`, which existed only to keep a copy fresh that nobody
reads now.

Event memory used a `_segment_uuid` property to reach a derivative's
segment. The segment store already holds that mapping on the derivative's
own row, non-null, under a primary key that leads on exactly the columns
the lookup filters -- so the copy bought nothing an indexed read does not,
and `get_segment_uuids_by_derivative_uuids` mirrors the forward lookup that
was already there. The property is gone, and with it a reserved field name.

Semantic memory had no mapping to reach for. Its record uuid was
`uuid5(namespace, feature_id)`, which is one-way, so the feature id had to
ride along in the properties. It now owns a `vector_uuid` column, unique
and minted per feature, and resolves hits through it -- the vector id is
the vector's own, and the caller keeps the correspondence. Every property
that collection carried was a copy of a column on the feature row, never
filtered on, so the payload goes entirely.

The delete paths read that column before deleting the rows, since the row
is what says which vector record a feature owns.

Data on speedkick does not survive this: existing semantic features have no
`vector_uuid`, and existing collections carry properties nothing reads.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Restore the Nebula tests a bad edit took with it

Cutting `test_similarity_metric_mappings` searched for the next `\ndef ` or
`\nclass ` to find where the function ended. Every test after it is `async
def`, so both searches missed, the end fell through to end-of-file, and the
edit deleted 429 lines: ten test functions, of which four had anything to
do with similarity metrics.

Gone with it were the empty-input cases for adding nodes and edges, the
none-property cases for both, the wrong-collection delete, the
nonexistent-uid read, the multi-property directional search -- and
`test_search_similar_nodes_cosine_metric`, the test for the one metric that
survives this change.

Nothing caught it because these tests skip everywhere, CI included: the
vector support they exercise needs NebulaGraph Enterprise >= 5.0, which the
fixture reaches at NEBULA_HOST and skips without.

Rebuilt from the file on speedkick with the metric stripping applied, then
removing only what has no subject left: the mapping helpers' test, the dot
and manhattan searches, and the ANN search -- Nebula indexes only L2 and
IP, so with cosine alone no index it can build serves a query.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Keep NebulaGraph's ANN by spelling cosine as an inner product

Nebula's `cosine()` is KNN-only -- it cannot take APPROXIMATE -- and its
vector indexes offer only L2 and IP. From that I concluded cosine could
never be approximate here and deleted the ANN search branch and vector
index creation as unreachable. That was wrong: cosine similarity between
unit vectors *is* their inner product, so an IP index over normalized
vectors gives cosine ranking with ANN.

Vectors are normalized in `_vector_to_gql_literal`, which is the single
place any vector becomes a literal -- node writes, edge writes, ANN queries
and exact queries all pass through it, so the stored side and the query
side cannot disagree about it. `inner_product()` DESC against an IP index
replaces the metric lookups.

IP rather than L2, though both rank identically over unit vectors
(‖a-b‖² = 2(1-cos), monotone in cos, and verified equal by argsort). The
reason is not numerical: measured over near-duplicate float32 unit vectors,
recovering cosine as 1 - d²/2 is 1.1x worse than reading it off IP, which
is nothing. It is that the contract answers a cosine similarity, and with
IP over unit vectors the index score already is one -- no conversion, no
assumption about whether the engine hands back the distance or its square,
and no result landing outside [-1, 1] needing a clamp.

`test_search_similar_nodes_ann` comes back with it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Drop the constant aliases left over from configurable metrics

Collapsing the metric enum left an instance attribute that only ever
copied a class constant (self._space = self._SPACE) and, in the Nebula
store, a local that copied a constant into a second local before use.
Read the constants where they are used.

The Nebula metric names move to module scope alongside the package's
other fixed identifier constants; they are neither per-instance nor
overridden.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit abf92a3)
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 25, 2026
… (speedkick) (MemMachine#1598)

* Answer with cosine scores, not with stored vectors

Two changes to one contract, taken together so its signatures are not
churned twice.

Cosine is now the only similarity. Every embedder MemMachine ships already
produced vectors meant to be compared that way -- OpenAI hard-coded it,
Bedrock defaulted to it, SentenceTransformer only reported what the model
declared -- while every layer that touched a score paid for the other
three metrics in direction flags, threshold directions, and per-backend
tables mapping the enum onto native metric names. `SimilarityMetric` is
gone; scores are cosine similarities in [-1, 1] and the names say so:
`QueryMatch.score` and `SearchMatch.score` become `cosine_similarity`,
`query(score_threshold=)` becomes `query(min_cosine_similarity=)`, which
no longer needs a direction to be meaningful. The Bedrock embedder's
`similarity_metric` config key goes with it, and the install and
configuration docs drop it.

And `VectorStoreCollection.get` is removed, with nothing offered in its
place. It had one production caller: the semantic storage read a feature's
stored embedding back so it could write the same embedding again with
fresh properties, because `upsert` demands a whole record.
`return_vector=True` was passed at that one call site and nowhere else,
and `VectorSearchEngine.get_vectors` existed to serve it. `set_properties`
serves that caller directly -- correcting a record's properties no longer
requires holding its vector -- and on SQLite and sqlite-vec it is an
UPDATE that never touches the index.

No scoring-by-id entry point takes `get`'s place. One would be needed if a
caller assembled a candidate set outside the store and asked for those ids
to be scored, which is what a selective-filter plan running above the
store would do. Property filtering stays inside the store instead, so the
candidate set stays there too, and a store-side regime can reach engine
keys directly without a public method addressed by record UUID.

Two consequences beyond the vector store. The vector graph stores carried
a metric per stored embedding, as a companion property beside every
vector; that is gone and `Node.embeddings` holds plain vectors.
NebulaGraph indexes only L2 and IP and its `cosine()` cannot be
APPROXIMATE, so with cosine alone no index it can build serves a query --
its ANN branch and vector index creation could no longer run and are
removed; search there is always exact KNN.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Let each store own its own mapping, and stop copying it

The vector store is not the authority for anything but vectors, so a
consumer that read a record's properties back out of it was reading a copy.
Both consumers did that, and for the same reason: to get from a search hit
back to a domain id. What the copy cost them differed. Semantic memory's
was of mutable columns, correct only as far as the last write to it and
stale from then on. Event memory's was an immutable uuid pairing, so it
stayed correct -- it simply duplicated a mapping the segment store already
served from an index, and spent a reserved property name doing it.

`QueryMatch` therefore answers `record_uuid` and a score. Properties stay
stored and filterable, because a filter is evaluated against the copy
rather than trusted as the record, and are no longer returned. That is the
end of `return_properties`, of `Record` on the read path, and of
`set_properties`, which existed only to keep a copy fresh that nobody
reads now.

Event memory used a `_segment_uuid` property to reach a derivative's
segment. The segment store already holds that mapping on the derivative's
own row, non-null, under a primary key that leads on exactly the columns
the lookup filters -- so the copy bought nothing an indexed read does not,
and `get_segment_uuids_by_derivative_uuids` mirrors the forward lookup that
was already there. The property is gone, and with it a reserved field name.

Semantic memory had no mapping to reach for. Its record uuid was
`uuid5(namespace, feature_id)`, which is one-way, so the feature id had to
ride along in the properties. It now owns a `vector_uuid` column, unique
and minted per feature, and resolves hits through it -- the vector id is
the vector's own, and the caller keeps the correspondence. Every property
that collection carried was a copy of a column on the feature row, never
filtered on, so the payload goes entirely.

The delete paths read that column before deleting the rows, since the row
is what says which vector record a feature owns.

Data on speedkick does not survive this: existing semantic features have no
`vector_uuid`, and existing collections carry properties nothing reads.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Restore the Nebula tests a bad edit took with it

Cutting `test_similarity_metric_mappings` searched for the next `\ndef ` or
`\nclass ` to find where the function ended. Every test after it is `async
def`, so both searches missed, the end fell through to end-of-file, and the
edit deleted 429 lines: ten test functions, of which four had anything to
do with similarity metrics.

Gone with it were the empty-input cases for adding nodes and edges, the
none-property cases for both, the wrong-collection delete, the
nonexistent-uid read, the multi-property directional search -- and
`test_search_similar_nodes_cosine_metric`, the test for the one metric that
survives this change.

Nothing caught it because these tests skip everywhere, CI included: the
vector support they exercise needs NebulaGraph Enterprise >= 5.0, which the
fixture reaches at NEBULA_HOST and skips without.

Rebuilt from the file on speedkick with the metric stripping applied, then
removing only what has no subject left: the mapping helpers' test, the dot
and manhattan searches, and the ANN search -- Nebula indexes only L2 and
IP, so with cosine alone no index it can build serves a query.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Keep NebulaGraph's ANN by spelling cosine as an inner product

Nebula's `cosine()` is KNN-only -- it cannot take APPROXIMATE -- and its
vector indexes offer only L2 and IP. From that I concluded cosine could
never be approximate here and deleted the ANN search branch and vector
index creation as unreachable. That was wrong: cosine similarity between
unit vectors *is* their inner product, so an IP index over normalized
vectors gives cosine ranking with ANN.

Vectors are normalized in `_vector_to_gql_literal`, which is the single
place any vector becomes a literal -- node writes, edge writes, ANN queries
and exact queries all pass through it, so the stored side and the query
side cannot disagree about it. `inner_product()` DESC against an IP index
replaces the metric lookups.

IP rather than L2, though both rank identically over unit vectors
(‖a-b‖² = 2(1-cos), monotone in cos, and verified equal by argsort). The
reason is not numerical: measured over near-duplicate float32 unit vectors,
recovering cosine as 1 - d²/2 is 1.1x worse than reading it off IP, which
is nothing. It is that the contract answers a cosine similarity, and with
IP over unit vectors the index score already is one -- no conversion, no
assumption about whether the engine hands back the distance or its square,
and no result landing outside [-1, 1] needing a clamp.

`test_search_similar_nodes_ann` comes back with it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Drop the constant aliases left over from configurable metrics

Collapsing the metric enum left an instance attribute that only ever
copied a class constant (self._space = self._SPACE) and, in the Nebula
store, a local that copied a constant into a second local before use.
Read the constants where they are used.

The Nebula metric names move to module scope alongside the package's
other fixed identifier constants; they are neither per-instance nor
overridden.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit abf92a3)
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 29, 2026
…ector

`get` had one production caller: the semantic storage's `update_feature`,
which read a feature's stored vector back so it could write it again
with fresh properties, since `upsert` replaces a whole record. Event and
declarative memory only search. That read was a bug (MemMachine#1721): the value
read is written back, so a stale read becomes a lasting wrong write.
- Two concurrent updates of one feature, one with a new embedding and
  one without, can write the old embedding back over the new one, on
  every backend and at every consistency level: nothing serializes them.
- The row is committed before the read, so a read that misses the vector
  record fails the update with the row updated and the vector not. On
  Milvus a read at the collection's Session level waits only for the
  reading process's own writes, so an update served by another process
  can miss the record or read an older vector.

`get` cannot be kept as a read that is safe to write from. That needs it
to reflect every write that returned before it began, from any process:
Qdrant gives that on a single node, a replicated Qdrant only with read
consistency settings the store does not make, and Milvus only at Strong,
whose wait under steady writes (to other tenants) measured a p99 of
3.0-3.5 s on the async client (Milvus 2.6.24, 4 CPUs). A weaker `get` is
the bug above. Stating the store's consistency in the contract, which the
next commit does, is simpler and holds on every backend once `query` is
the only read.

Nothing replaces it. The feature row is the authority for everything but
the embedding. The semantic vector record now carries only `feature_id`,
the property a search hit is resolved through; the other ten were copies
of row columns nothing filtered on, and the collection declares no
indexed properties. `update_feature` writes the vector store only when
given a new embedding, and writes the whole record. MemMachine#1663 (the port of
MemMachine#1598) removes `get` the same way, with a `vector_uuid` column in place
of the property; this takes only what the consistency contract needs.

Tests that read a record back through `get` now read what the backend
holds past the store: the SQLite stores' records tables, a Qdrant scroll
on the incarnation, a Strong read on the Milvus client (the collection
lifecycle contract gains `stored_uuids` for this); a test whose subject is
what a query returns queries. Tests of `get` itself go. Three semantic
tests fail on the code before this commit: a vector record carries only
the feature id; an update without an embedding leaves the record alone;
an update reads nothing back, succeeding with the record absent.

Fixes MemMachine#1721.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 29, 2026
A query returned each match's record: its properties, and its vector on
request. They were copies of what the callers' own stores hold, only as
fresh as the store's reads, which may lag its writes, and returning them
invites a caller to treat the copy as the record. Nothing needed them:
- semantic memory read one property, `feature_id`, to resolve a hit to its
  row; it never filtered on the others, which copied row columns;
- event memory read one, `_segment_uuid`, to resolve a hit to its segment,
  a mapping the segment store already holds on the derivative's row;
- no caller asked for a vector.

`QueryMatch` now carries a score and a `record_uuid`, `query` loses
`return_vector` and `return_properties`, and `Record` is input-only.
Properties are still stored and filtered on. Each caller resolves a hit
through the store that owns the mapping:
- event memory through the segment store's new
  `get_segment_uuids_by_derivative_uuids`, served by the derivative
  table's primary key; its vector records no longer carry `_segment_uuid`;
- semantic memory through a `vector_uuid` column on the feature row, which
  its vector record is keyed by; the delete paths read the column before
  deleting the rows. The record carries no properties.
A hit whose segment or feature is gone is dropped.

What only served returned records goes: Milvus's `_tz_` offset fields,
which kept a declared datetime's timezone for reading it back; the stores'
record parsing; `VectorSearchEngine.get_vectors`; sqlite-vec's vector
decoding. This is the part of MemMachine#1598 (MemMachine#1663 on main) the consistency
contract needs; MemMachine#1663 keeps its cosine-only scores.

Breaking: a semantic feature table created before this has no
`vector_uuid` column, and the table is created with `create_all`, which
adds none. No migration; pre-GA, as with the rest of this PR's layouts.

Tests of return flags and of values read back through a query go; tests
that check what a store holds read the backend past the store, as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Oct 10, 2026
Every embedder MemMachine ships produces vectors meant to be compared by
cosine -- OpenAI hard-coded it, Bedrock defaulted to it, SentenceTransformer
only reported what the model declared -- while every layer that touched a
score paid for the other three metrics in direction flags, threshold
directions, and per-backend tables mapping the enum onto native metric
names. `SimilarityMetric` is gone; scores are cosine similarities in
[-1, 1] and the names say so: `QueryMatch.score` and `SearchMatch.score`
become `cosine_similarity`, and `query(score_threshold=)` becomes
`query(min_cosine_similarity=)`, which no longer needs a direction to be
meaningful and is refused when not finite, as the threshold was. A store
and its partitions no longer have a `similarity_metric`, the schema a
partition is registered under no longer records one, and a search engine
factory takes the dimensions alone. The Bedrock embedder's
`similarity_metric` config key and semantic memory's
`vector_similarity_metric` go with it, and the install and configuration
docs drop them.

The vector graph stores carried a metric per stored embedding, as a
companion property beside every vector; that is gone and `Node.embeddings`
holds plain vectors. NebulaGraph's `cosine()` cannot take `APPROXIMATE` and
its vector indexes offer only L2 and IP. Cosine similarity between unit
vectors is their inner product, so embeddings are normalized on the way in
and compared with `inner_product()` against an IP index.

The cosine half of MemMachine#1598 (`abf92a3a4` on speedkick), re-derived on the
one-collection store: the rest of MemMachine#1598 (queries answer record UUIDs and
scores, no `get`, semantic memory's `vector_uuid`) and all of MemMachine#1603 are in
registry-backed base and its partition handle lose the metric, the stores'
own threshold checks become `require_valid_min_cosine_similarity`, and the
design documents describe cosine scoring and a schema without a metric.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant