Repository navigation
Refuse a non-positive top_k at the schema - #1740
Merged
Merged
Conversation
top_k was declared int with a default of 10 and no lower bound, so 0 and -1 passed validation and reached retrieval, which cannot use them. The Qdrant path answered 500 with a bare "Internal Server Error" body -- nothing caught it, so nothing logged it through the API's exception handler either. A bound rather than a guard, because the two guards that already exist disagree about what a non-positive limit means: agent_api treats <= 0 as "no limit" and returns everything, sqlite_vector_store treats it as "no results" and returns nothing. Picking either would make the other wrong; refusing the input makes neither reachable, and top_k is documented as "the maximum number of memories to return", under which -1 has no reading at all. Found by mm-test functional FV-30. Signed-off-by: Haiyan Wang <[email protected]>
ruff format wanted the two long lines wrapped, and ruff check flagged the unescaped "|" in the pytest.raises match pattern (RUF043). Signed-off-by: Haiyan Wang <[email protected]>
edwinyyyu
approved these changes
Oct 2, 2026
leomem
reviewed
Oct 2, 2026
leomem
left a comment
Collaborator
There was a problem hiding this comment.
No issues found. The ge=1 bound on SearchMemoriesSpec.top_k works with every caller I checked: the Python client sends limit or 10, the MCP tool defaults to 20, the TS client to 10, and Dify and Strands send positive defaults. On the MCP path the ValidationError is caught and returned as a 422 McpResponse.
One behaviour change worth knowing: a negative limit passed to the Python client now raises a ValidationError on the client, before any request is sent.
malatewang
approved these changes
Oct 5, 2026
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Oct 7, 2026
…UUIDs and scores, and refuse invalid inputs (MemMachine#1733) * Look up the segments that derivatives belong to in the segment store Event memory resolves a search hit, a derivative's vector record, to the derivative's segment through a `_segment_uuid` property it copies onto every vector record. The segment store already holds that mapping, on the derivative's own row. `SegmentStorePartition.get_segment_uuids_by_derivative_uuids` reads it. The SQLAlchemy partition answers with one query on the derivative table, served by its primary key, and checks the partition's liveness in the same statement, as its other reads do. A derivative the partition does not hold is left out of the result. The in-memory partition the event memory tests use implements it too. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Resolve event memory's search hits through the segment store A derivative's vector record carried its segment's UUID as the `_segment_uuid` property, and a search read it back from each match. That copy is only as fresh as the vector store's reads, and the segment store holds the same mapping on the derivative's row. A search now asks the vector store for no properties and maps the matched derivatives to their segments with one `get_segment_uuids_by_derivative_uuids` call. A match whose derivative the segment store no longer holds, because its segment was deleted and its vector outlived it, is dropped. Vector records no longer carry `_segment_uuid`, and the schema event memory expects of its collection no longer declares it; a collection created with it keeps declaring it. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Resolve semantic search hits through a vector_uuid column on the feature row A feature's vector record was keyed by a UUID derived from the feature's id, and carried eleven properties copied from the feature's row, among them `feature_id`, which a search read back to resolve a hit to its row. The caller's feature metadata was merged into the same properties. - `update_feature` read the stored vector back through `VectorStoreCollection.get` to write the record again with fresh properties, since an upsert replaces a whole record. The value read is written back, so a stale read becomes a lasting wrong write: of two concurrent updates of one feature, one with a new embedding and one without, the second can write the old embedding over the new one; and a read that misses the record fails the update after its row was committed (MemMachine#1721). - Nothing filtered on the copied properties; the row is their authority. The feature row now carries `vector_uuid`, a fresh UUID that keys its vector record, and the record carries no properties. A search resolves its hits through that column, in the order the search returned them, dropping a hit whose row is gone. `update_feature` writes the vector store only when given a new embedding, and reads nothing back. The delete paths take the UUIDs of exactly the rows they delete with RETURNING. The collection declares no indexed properties. Breaking: a feature table created before this has no `vector_uuid` column, and the table is created with `create_all`, which adds none; and a collection created with the old declared properties no longer matches the configuration `open_or_create_collection` asks for. No migration is included. Fixes MemMachine#1721. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Answer a vector store query with record UUIDs and scores; remove get and close_collection A query returned each match's record: its properties, and its vector on request. They were copies of what the callers' own stores hold, only as fresh as the vector store's reads, and since the previous two commits no caller reads them. `get` has had no caller since semantic memory stopped reading its vectors back, and a `get` safe to write from would need every backend to reflect every write that returned before it, from any process. Every store's `close_collection` did nothing, and nothing called it. - `QueryMatch` carries a score and a `record_uuid`, and `query` loses `return_vector` and `return_properties`. Properties are still stored and filtered on. - `VectorStoreCollection.get` and `VectorStore.close_collection` go, with what only served them: the stores' record parsing, `VectorSearchEngine.get_vectors`, and sqlite-vec's vector decoding. - `Record` is input-only, so its vector is required and its properties default to `{}`. The model rejects a missing vector at construction, so the stores' checks for one and their coalescing of `None` properties go. Mechanical: event memory and semantic memory read `match.record_uuid` in place of `match.record.uuid` and stop passing the flags, and the in-memory test collection follows the interface. Tests of the return flags, of `get`, and of values read back through a query go. Tests that check what a store holds read the backend past the store: the SQLite stores' records tables, a Qdrant scroll of the collection's partition, a Milvus `get` on the client. The tests of a store refusing a `None` vector become tests that the model refuses a missing one, plus one that properties default to `{}` and one that a record without properties is stored. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Refuse vector store inputs no collection can hold or search Each store refuses, with a ValueError and before any backend call: - a record whose declared property holds a value of another type, since the collection indexes and filters the property as its declared type (`require_declared_types`: the value's type is the declared type, and a bool is not an int); - a record vector or a query vector whose width is not the collection's dimensions; - a query vector with a coordinate that is not finite, or a score threshold that is not finite. The model checks a record's own vector for finiteness: `Record.vector` is `list[FiniteFloat]`, so pydantic refuses a NaN or infinite coordinate when the record is built, in the pass that already validates each coordinate. A query vector is a plain sequence, so the stores check it. An embedding endpoint can return NaN whatever the caller does, so both kinds of vector are checked. Qdrant and Milvus check a filter's property keys with the other inputs, before the early return for no query vectors. `validate_identifier` uses `fullmatch`: `$` also matches before a trailing newline, so `"name\n"` passed as a namespace, a collection name or a property key. The ABC states these refusals in `upsert` and `query`. Tests: each store refuses each input, and a record refuses a coordinate that is not finite. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * State the vector store's consistency in its contract The contract said nothing about what a query sees of earlier writes. `VectorStoreCollection` now states what every store keeps: An `upsert` or `delete` is durable once it returns; queries may not reflect it right away. A store that guarantees more states it. Neither event memory nor semantic memory reads its own writes back through the vector store: a search's hits resolve through the segment store or the feature row, which hold the mapping. Milvus reads at the consistency level its collections are configured with, and a replicated Qdrant may answer a query from a replica that has not applied a write yet, so a stronger promise would bind every store to read settings with costs of their own. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Name the derivative-to-segment map segments_by_derivatives Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Drop a None check no property reaches, and a filter branch no caller takes - The Qdrant store's payload skipped a None property, but a Record's properties are PropertyValues, which exclude None, so the model refuses one before the store sees it. - Semantic storage's _apply_feature_filter took a Select or a Delete, but only selects reach it: delete_feature_set filters its DELETE ... RETURNING itself. It takes and answers a Select, and the cast at its caller goes. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Tidy the tests the review found out of place - The SQLite stores' four input-validation tests sat inside TestFilters, under a "Delete" banner meant for TestDelete; they move into TestInputValidation, and the banner moves above TestDelete. - Two event memory schemas still declared `_segment_uuid`, which event memory no longer reserves: the context test declares `_timestamp` alone, and the missing-base-field test declares nothing. - The 40,000-feature test's comment says which SQLite the bind limit is the feature store's. Tests only; the same tests run and pass. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Fold the declared-type check into the vector store utilities declared_properties.py held one function beside utils.py, which holds the other input checks every store runs (dimensions, query vector, score threshold, identifiers). It imports only common.data_types, which imports nothing back, so it moves into utils.py as is. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Resolve semantic search hits in one statement Since queries answer record UUIDs alone, a semantic search mapped its hits to feature ids in one select, then loaded the features by id with the filter in another: two statements in two sessions, and the first ran even when the search found nothing. One select on the vector_uuid column with the filter now loads the features, ordered as the hits were, and no select runs for a search with no hits. Results are unchanged: the same features, in the same order, under the same filter. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Refuse a query limit that is not positive The contract said nothing of a limit at or below zero, and the stores disagreed: the SQLite stores and Milvus answered empty results, and Qdrant passed the limit on for the server to reject. A limit that is not positive is now a ValueError in every store, stated in the query contract beside the other refused inputs, and checked whether or not there are query vectors. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Refuse a search for no memories before it does any work A search whose top_k is not positive asks for nothing, yet it reached the vector store, which refuses a limit that is not positive, only after event memory had embedded the query: a request for nothing still cost an embedding call. SearchMemoriesSpec.top_k is now positive (gt=0), so request validation refuses it with a 422 naming top_k, the MCP tool included, and LongTermMemory.search_scored refuses a num_episodes_limit that is not positive before the query is embedded, for callers of the Python API. The event backend's window test no longer runs a limit of 0 on the in-memory vector fake, which does not check limits; its lower bound is now exercised with a negative expand_context. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Show the positive top_k in the OpenAPI document docs/openapi.json, as docs/tools/generate_openapi.py writes it for the search specification's top_k: exclusiveMinimum 0 and the description. The generator's other differences under the locked FastAPI are left to the regeneration that carries them. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Rebased onto main. Main refuses a top_k that is not positive since MemMachine#1740, with ge=1 and its own tests, so this commit keeps that bound and those tests in place of the gt=0 bound and the test described above, and docs/openapi.json shows minimum 1. vector_store_semantic_storage.py keeps importing datetime, which the history methods MemMachine#1707 added use.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose of the change
Refuse a non-positive
top_kon search at the schema, so the server answers422 instead of 500.
Description
SearchMemoriesSpec.top_kwas anintwith a default of 10 and no lowerbound, so
0and-1passed validation and reached retrieval, which cannotuse them. On the Qdrant path the request failed with a 500 and a bare
"Internal Server Error" body. Nothing caught the error, so the API's exception
handler never logged it.
This adds
ge=1to the field. The search route takesSearchMemoriesSpecasits body, so FastAPI now returns a 422 that names the field.
I chose a bound over a guard because the guards that already exist further
down disagree about what a non-positive limit means:
retrieval_agent/common/agent_api.pytreats<= 0as "no limit" andreturns everything.
return nothing.
Picking either meaning would make the other wrong. Refusing the input means
neither path is reached.
top_kis documented as "the maximum number ofmemories to return", and
-1has no meaning under that.Other callers:
search_memorybuilds the spec inside its existingtry, so a badtop_kcomes back as a 422McpResponse.top_k=limit or 10, solimit=0still becomes 10.limit=-1now raises a pydanticValidationErroron the client before anyrequest is sent.
Found by mm-test functional FV-30.
Fixes/Closes
No issue filed.
Type of change
Clients that sent
top_k <= 0used to get a 500 (Qdrant) or a backend-specificresult. They now get a 422.
How Has This Been Tested?
New tests in
server/api_v2/test_spec.py:test_search_top_k_must_be_positive[0]and[-1]: validation is refused.test_search_top_k_accepts_one_and_the_default:1and the default10are still accepted.
Test Results:
uv run pytest packages/server/server_tests/memmachine_server/server/api_v2/test_spec.pygives 55 passed.
ruff format --checkandruff checkare clean on thechanged files.
Checklist