Skip to content

Refuse a filter value of the wrong type for a datetime column, and answer an invalid list filter with 422 (port of #1620) - #1786

Draft
edwinyyyu wants to merge 1 commit into
MemMachine:mainfrom
edwinyyyu:port/filter-datetime-column-typing-main
Draft

edwinyyyu wants to merge 1 commit into
MemMachine:mainfrom
edwinyyyu:port/filter-datetime-column-typing-main

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1619.

Port

Copy of #1620, merged into speedkick as 8d7b832a1, onto main, which has the same parser, compiler, and route, so the same defect. The one conflict is in list_memories: main lists every memory type when a request names none, and the port keeps that default inside #1620's 422 mapping.

Purpose of the change

A filter such as created_at > '2026-01-01T00:00:00Z' reached PostgreSQL as timestamp with time zone > character varying, failed there, and came back as HTTP 500 from /memories/list (episodic and semantic) and from /memories/search with types: ["semantic"]. The same filter ran on SQLite as a text comparison. SQLAlchemy types a bind by its Python value, so the mismatch was invisible until the database rejected it.

Three changes, one per concern, in one commit here as #1620 merged squashed:

  1. compile_sql_filter refuses the pairing before a statement is built. A column-encoded leaf pairs a DateTime column with a datetime value and nothing else; a str, an int, or an IN list against created_at (or timestamp on the segment store), and a date() literal against a non-datetime column, raise ValueError with a message that names the field and points at date('...'). This holds on every SQL-backed store that resolves a datetime column: the episode store, the segment store, and both semantic feature stores.
  2. list_memories maps ValueError to 422, as search_memories already did. Before, every invalid filter on list (a syntax error, an unknown field, this mismatch) was a 500.
  3. The filter field's description (SpecDoc.FILTER_MEM, and the two copies in docs/openapi.json) names date('...') as the form a datetime field compares with. Only those two descriptions change in the spec; a full local regeneration also drifts on ValidationError and info.version, which the main workflow owns.

The value is rejected rather than coerced: the filter language types a value by its literal, date() is the datetime literal, and a properties_json leaf already matches only a value of the declared type. Coercing a string here would make the column encoding disagree with the others.

Relation to #1616: that PR makes the parser reject a string in an ordering comparison, which covers >/</>=/<= with a str. This change is at the column boundary, so it also covers =, !=, an integer, and an IN list, and it does not depend on the vector store stack. The two compose; the rebase of either over the other is a few lines in _compile_column_leaf.

Stack

21 open PRs: three independent PRs, and the vector store tree of short parallel branches. Every PR in the tree has feat/horizontal-scaling as its GitHub base, and the independent PRs have main. The branches are in a fork, and a pull request can target only this repository's branches, so the on column gives the order the PRs build on each other. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

# PR change on
— #1624 Make no memory request create a project main
— #1786 (this PR) Refuse a filter value of the wrong type for a datetime column, and answer an invalid list filter with 422 (port of #1620) main
— #1792 Refuse property values that some store refuses or alters where an episode enters main

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1702 and #1670 sit beneath #1627, whose code depends on them. Until the PRs under it merge, their changes show in a stacked PR's diff.

# PR change on
[vector store scale-out 1/6] #1671 (merged) Remove custom sharding from the Qdrant store (port of #1654) main
[vector store scale-out 2/6] #1733 (merged into feat/horizontal-scaling) Answer vector store queries with record UUIDs and scores, and refuse invalid inputs feat/horizontal-scaling
[vector store scale-out 3/6] #1734 (merged into feat/horizontal-scaling) Arbitrate vector store collections in a SQL registry, with an incarnation per collection life and a purge feat/horizontal-scaling
[vector store scale-out 4/6] #1735 (merged into feat/horizontal-scaling) Move the Qdrant store onto the collection registry feat/horizontal-scaling
[vector store scale-out 5/6] #1736 (merged into feat/horizontal-scaling) Move the Milvus store onto the collection registry, against a Milvus server feat/horizontal-scaling
— #1775 (merged into feat/horizontal-scaling) Accept attempts exhausted in the lifecycle churn contract, and say which Qdrant operations filter on the incarnation feat/horizontal-scaling
— #1779 (merged into feat/horizontal-scaling) Classify Qdrant errors by status code alone feat/horizontal-scaling
— #1813 (merged into feat/horizontal-scaling) Create Qdrant collections with strict mode off feat/horizontal-scaling
— #1788 Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract feat/horizontal-scaling
— #1631 (closed) Superseded by #1733–#1736, which hold its changes split in four, with review changes since —
[user properties 1/2] #1702 Keep undeclared properties out of the vector store feat/horizontal-scaling
[user properties 2/2] #1670 Remove per-project filterable properties (port of #1606) #1702
[vector store scale-out 6/6] #1627 Make a vector store one collection, with string-keyed partitions #1670
[session storage 1/2] #1622 Create a session's storage with the session, never on a request #1627
[session storage 2/2] #1625 Remove open-or-create from both stores, and close from the segment store #1622
[declared schema 1/2] #1628 Make a vector store filter only on the properties it declares #1627
[search results] #1663 Score every vector search by cosine similarity, and name scores for it (port of #1598's cosine half) #1628
[declared schema 2/2] #1616 Close the filter union, and make negation the complement on every backend #1663
[sqlite store fixes 1/7] #1460 Publish vector index files atomically (but not durably) #1663
[sqlite store fixes 2/7] #1469 Never reuse a row id in SQLiteVectorStore #1460
[sqlite store fixes 3/7] #1672 Own the search engine's concurrency in the store, not in each engine (port of #1612) #1469
[sqlite store fixes 4/7] #1673 Serialize a partition's writes so the engine sees them in order (port of #1607) #1672
[sqlite store fixes 5/7] #1674 Refuse a pending row replay cannot honor, instead of dropping it (port of #1608) #1673
[sqlite store fixes 6/7] #1675 Take SQLite's write lock at BEGIN, not at the first write (port of #1609) #1674
[sqlite store fixes 7/7] #1676 Give every write a fresh row id, so a key names one version (port of #1610) #1675
[qdrant options] #1618 Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization #1663
[milvus options] #1741 Let a deployment tune a Milvus collection's vector index and its searches #1618

This PR is its one commit, 899719556, directly on main, independent of the vector store tree: review and merge it in any order.

Verification

As measured on speedkick for #1620:

  • test_sql_filter_util.py: the fixture model gains a DateTime column; four mismatched filters raise with "holds a datetime", name > date(...) raises with "does not hold a datetime", and created_at > date(...) / >= date(...) compile and select the expected rows. test_bounds_are_normalized_to_utc's fixture column becomes a DateTime, which is what it models.
  • test_episode_storage.py: created_at > '...', created_at = '...' and created_at > 5 raise ValueError from both get_episode_messages and get_episode_messages_count, on SQLite and (under -m integration) PostgreSQL.
  • test_router.py: /memories/list answers a ValueError from list_search with 422.

uv run pytest packages/server/server_tests: 1937 passed, 3 skipped. -m integration on test_episode_storage.py (PostgreSQL via testcontainers): 24 passed. ruff check, ruff format --check, and ty check --project packages/server clean.

End to end on a server backed by pgvector/pgvector:pg16 (pool_size: 2, max_overflow: 0): the text form on list (episodic, semantic) and on semantic search answers 422 with Field 'created_at' holds a datetime; compare it with date('...'), not str; nine of them in a row leave pg_stat_activity unchanged and the next valid request answers 200 in 4 ms. The same run on the unpatched tree answers 500 at each of those points. The connection leak in the original report did not reproduce on either tree; details in #1619.

The port, on main at ad8ff24b0: ruff check, ruff format --check, and ty check are clean, and uv lock --check passes; the full server suite without integration tests passes (2114 passed, 3 skipped); in test containers, the episode store's, the filter compiler's, and the server routes' integration tests pass against PostgreSQL (92 passed). It merges with feat/horizontal-scaling without conflicts.

🤖 Generated with Claude Code

…swer an invalid list filter with 422 (port of MemMachine#1620)

* Refuse a filter value of the wrong type for a datetime column before the statement is built (speedkick)

A column-encoded leaf now pairs a DateTime column with a datetime value
and nothing else; any other pairing raises ValueError in
compile_sql_filter, on every SQL-backed store, with a message that
names the field and points at date('...').

SQLAlchemy types a bind by its Python value, not by the column, so
created_at > '2026-01-01T00:00:00Z' reached PostgreSQL as
timestamp with time zone > character varying and failed as a
ProgrammingError, which the routes answered with 500; on SQLite the
same statement ran as a text comparison. The reverse pairing, a date()
literal against a non-datetime column, failed the same way.

Tests: the compile-level cases (str, int and IN against a datetime
column; date() against a string column; the ordering that does
compile) and the episode store's created_at filter on both engines.
The normalization test's fixture column becomes a DateTime, which is
what it models.

Refs MemMachine#1619

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01FNHipcrwW9JByUwhg3JVcj

* Answer an invalid filter on /memories/list with 422, as /memories/search does (speedkick)

list_memories mapped no error to a status, so a FilterParseError, an
unknown field or a value of the wrong type was a 500 there and a 422
on search. The same ValueError -> 422 mapping now applies to both.

Refs MemMachine#1619

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01FNHipcrwW9JByUwhg3JVcj

* Document the date() literal on the filter field (speedkick)

The filter description names date('...') as the form a datetime field
such as created_at compares with, and says a quoted string or a number
there is an invalid argument. docs/openapi.json carries the same two
descriptions; the rest of a local regeneration is drift the main
workflow owns.

Refs MemMachine#1619

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01FNHipcrwW9JByUwhg3JVcj

---------

Ported onto main from 8d7b832 on speedkick. /memories/list on main
lists every memory type when none is given; the port keeps that default
and maps a ValueError from listing to 422 around it.

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Co-authored-by: Shu Wang <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This was referenced Oct 7, 2026
@edwinyyyu
edwinyyyu marked this pull request as draft October 7, 2026 20:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: A filter comparing created_at with a non-date() value returns HTTP 500 on PostgreSQL; /memories/list returns 500 for every invalid filter

1 participant