Repository navigation
[vector store 2/12] Route system filters to the vector store and user-defined ones to the segment store (speedkick) - #1621
Closed
edwinyyyu wants to merge 3 commits into
Conversation
This was referenced Sep 14, 2026
[session storage 2/2] Remove open-or-create from both stores, and close from the segment store
#1625
Draft
Closed
Draft
[qdrant options] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization
#1618
Draft
edwinyyyu
force-pushed
the
feat/event-memory-filter-routing-speedkick
branch
from
September 14, 2026 21:49
bb85596 to
b37d6b3
Compare
…egment store EventMemory split a caller's filter nowhere: the whole tree went to the vector store, which had to store every caller property and filter on any key, and then to the segment store again. Now the vector store gets the conjuncts naming only keys its collection declares, evaluated during the search, and the rest is the segment store's, applied to the seeds afterward; a disjunction or negation mixing declared and undeclared keys is undeclared as a whole. A derivative record carries only the caller properties the collection declares, so an undeclared key never reaches the vector store, and a declared key's value must have the declared type. A post-filter can drop seeds and leave the search short, so the fetch is widened by a factor of four at a time until the limit is met or the fetch reaches `limit * FilterOptions.max_overfetch_factor` (default 64), where the search returns what survived. A query with no undeclared part never widens. `filter_parser` gains `filter_fields`, `filter_nodes` and `split_declared`; the in-memory test collection records the filter of each query so routing can be asserted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
edwinyyyu
force-pushed
the
feat/event-memory-filter-routing-speedkick
branch
from
September 14, 2026 21:54
b37d6b3 to
eaf0133
Compare
…options type `FilterOptions` held a single integer and nothing read it but EventMemory; a container around one knob is structure without a second member. `max_overfetch_factor` is a field of EventMemoryParams, with the same default and bound. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…ening A search widened its vector fetch by four at a time, up to `max_overfetch_factor` times the limit, to make up for seeds the segment store's post-filter dropped, so a selective predicate on an undeclared key could cost 64 times the limit in vectors and segment store lookups. The vector search is one fetch of `vector_search_limit`, as it was: that limit bounds a query's cost, and the deployment declares the keys it filters on so they are evaluated during the search. A predicate on an undeclared key is applied to those seeds afterward, and the result holds the seeds it admits, fewer than the limit when it drops some; the docstring says so. The loop, its two constants and `max_overfetch_factor` go. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
edwinyyyu
changed the title
edwinyyyu
force-pushed
the
feat/event-memory-filter-routing-speedkick
branch
from
September 14, 2026 23:21
07b0b21 to
27724f6
Compare
edwinyyyu
marked this pull request as draft
September 14, 2026 23:39
Contributor
Author
|
Superseded by modifying #1597 |
Contributor
Author
|
Closed: #1597 stands in for this slice. Its typed query parameters (since/until/session_ids/source_ids/block_kinds) are the system filters the vector store evaluates, and |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose of the change
EventMemory took one filter tree and split it by the store's declared keys, so what the vector store evaluated depended on the store's schema: a user-defined key a deployment declared was evaluated in the vector store, against the decision that user-defined properties are never indexed there.
EventMemory.query now takes two filters.
system_filternames system fields by their bare names (timestamp, and the fields its caller writes under the_-prefixed layout); it is evaluated during the vector search and again by the segment store.property_filternames user-defined properties (m.<key>) only; the vector store never holds a user-defined property, so it is applied by the segment store to the seeds the one fetch returned, conjoined with the system filter. A bare name inproperty_filteror am.<key>insystem_filteris refused, and a vector record carries the system properties and never a user-defined one.The API still carries one filter, so the long-term-memory adapter takes it apart by field namespace with
split_conjunctsand refuses a clause that mixes the two under one disjunction or negation. EventMemory itself splits nothing.Stack
Slice 2 of 12, every PR targeting
speedkick; merge bottom-up.This PR's own change is its four commits:
eaf0133f(the routing, first cut), then on review34fa10de(folds the one-fieldFilterOptionsinto a plain field),27724f6d(removes the widening loop and the field: one fetch ofvector_search_limit) and07b0b218(6 files changed, 240 insertions(+), 89 deletions(-); EventMemory takessystem_filterandproperty_filterapart, evaluates only the former in the vector store, and the adapter splits the API's one filter by namespace); it sits directly onspeedkick, and #1622 is stacked on it.Verification
ruff check,ruff format --check,ty check(two pre-existing spacy diagnostics),pytest packages/server/server_tests packages/client/client_tests: 2213 passed, 3 skipped, on the slice-10 tree, 2026-09-14; every slice passed its own run when built.🤖 Generated with Claude Code
https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn