[user properties 2/2] Remove per-project filterable properties (port of #1606) by edwinyyyu · Pull Request #1670 · MemMachine/MemMachine

edwinyyyu · 2026-09-16T23:42:30Z

Port

Copy of #1606, merged into speedkick as a8322a790, on #1702, beneath #1627, whose code depends on it: the same commit, with the per-project schema also removed from #1736's service locator, which creates the session's collection in a retry loop, and the commented option removed from the event sample configuration #1698 added. The squash also carries #1606's side commit regenerating docs/openapi.json under the locked FastAPI (ValidationError gains input and ctx), seven lines of that file's diff; the rest of it removes properties_schema.

Purpose of the change

A project could declare properties_schema, a set of caller property keys with types, on its long-term memory configuration; the event backend merged it into the vector store collection's indexed schema and rejected filters on any other m.<key>. That let a tenant create database resources (indexes, columns) by naming them in a request, which is what forced per-collection native resources named by a hash of their schema on the backends that limit them.

The option is removed from the server configuration, the project API and the memory-configuration API, the Python SDK, the sample configurations, the configuration docs and the OpenAPI document. A filter may name any m.<key>; #1702, beneath this, keeps such keys off the vector store, so the segment store evaluates it, and with the option gone no tenant key reaches the vector store at all. What a store indexes is decided by the deployment, not per project.

A breaking API change on main: properties_schema leaves ProjectConfig, the SDK's create_project and get_or_create_project, and the memory-configuration update route.

Part of the fix for #1781.

Stack

21 open PRs: three independent PRs, and the vector store tree of short parallel branches. Every PR in the tree has feat/horizontal-scaling as its GitHub base, and the independent PRs have main. The branches are in a fork, and a pull request can target only this repository's branches, so the on column gives the order the PRs build on each other. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

#	PR	change	on
—	#1624	Make no memory request create a project	`main`
—	#1786	Refuse a filter value of the wrong type for a datetime column, and answer an invalid list filter with 422 (port of #1620)	`main`
—	#1792	Refuse property values that some store refuses or alters where an episode enters	`main`

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1702 and #1670 sit beneath #1627, whose code depends on them. Until the PRs under it merge, their changes show in a stacked PR's diff.

#	PR	change	on
[vector store scale-out 1/6]	#1671 (merged)	Remove custom sharding from the Qdrant store (port of #1654)	`main`
[vector store scale-out 2/6]	#1733 (merged into `feat/horizontal-scaling`)	Answer vector store queries with record UUIDs and scores, and refuse invalid inputs	`feat/horizontal-scaling`
[vector store scale-out 3/6]	#1734 (merged into `feat/horizontal-scaling`)	Arbitrate vector store collections in a SQL registry, with an incarnation per collection life and a purge	`feat/horizontal-scaling`
[vector store scale-out 4/6]	#1735 (merged into `feat/horizontal-scaling`)	Move the Qdrant store onto the collection registry	`feat/horizontal-scaling`
[vector store scale-out 5/6]	#1736 (merged into `feat/horizontal-scaling`)	Move the Milvus store onto the collection registry, against a Milvus server	`feat/horizontal-scaling`
—	#1775 (merged into `feat/horizontal-scaling`)	Accept attempts exhausted in the lifecycle churn contract, and say which Qdrant operations filter on the incarnation	`feat/horizontal-scaling`
—	#1779 (merged into `feat/horizontal-scaling`)	Classify Qdrant errors by status code alone	`feat/horizontal-scaling`
—	#1813 (merged into `feat/horizontal-scaling`)	Create Qdrant collections with strict mode off	`feat/horizontal-scaling`
—	#1788	Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract	`feat/horizontal-scaling`
—	#1631 (closed)	Superseded by #1733–#1736, which hold its changes split in four, with review changes since	—
[user properties 1/2]	#1702	Keep undeclared properties out of the vector store	`feat/horizontal-scaling`
[user properties 2/2]	#1670 (this PR)	Remove per-project filterable properties (port of #1606)	#1702
[vector store scale-out 6/6]	#1627	Make a vector store one collection, with string-keyed partitions	#1670
[session storage 1/2]	#1622	Create a session's storage with the session, never on a request	#1627
[session storage 2/2]	#1625	Remove open-or-create from both stores, and close from the segment store	#1622
[declared schema 1/2]	#1628	Make a vector store filter only on the properties it declares	#1627
[search results]	#1663	Score every vector search by cosine similarity, and name scores for it (port of #1598's cosine half)	#1628
[declared schema 2/2]	#1616	Close the filter union, and make negation the complement on every backend	#1663
[sqlite store fixes 1/7]	#1460	Publish vector index files atomically (but not durably)	#1663
[sqlite store fixes 2/7]	#1469	Never reuse a row id in SQLiteVectorStore	#1460
[sqlite store fixes 3/7]	#1672	Own the search engine's concurrency in the store, not in each engine (port of #1612)	#1469
[sqlite store fixes 4/7]	#1673	Serialize a partition's writes so the engine sees them in order (port of #1607)	#1672
[sqlite store fixes 5/7]	#1674	Refuse a pending row replay cannot honor, instead of dropping it (port of #1608)	#1673
[sqlite store fixes 6/7]	#1675	Take SQLite's write lock at BEGIN, not at the first write (port of #1609)	#1674
[sqlite store fixes 7/7]	#1676	Give every write a fresh row id, so a key names one version (port of #1610)	#1675
[qdrant options]	#1618	Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization	#1663
[milvus options]	#1741	Let a deployment tune a Milvus collection's vector index and its searches	#1618

This PR is its one commit, 799282992, stacked on #1702. #1627 is stacked on it.

Verification

At this PR's head 799282992, on 2026-10-09, on feat/horizontal-scaling with #1813 merged: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean.

Before the rebase onto #1813's merge: At this PR's head 587373950, on 2026-10-09: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); uv lock --check clean. The full suites were not run at this head: the server suite without integration tests last passed at 9031d25f4, on 2026-10-08, 2166 tests, and the integration tests of the vector stores, the resource manager, episodic memory, and semantic storage at 9031d25f4, on 2026-10-08, 689 tests, against PostgreSQL 16, Neo4j, Qdrant 1.19.1, and Milvus 2.6.24 in containers. This head differs from those by #1779's change to the Qdrant store and the rebase onto the merged #1736.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

The event backend wrote every property of an event into its vector record and mapped the caller's whole filter onto the vector store, so a user key that a deployment never declared was both stored and filtered there: on Qdrant and Milvus as unindexed payload a filtered query scans for. The segment store already holds every property and already receives the whole filter for the context windows, so the vector side only duplicated work the segment store does anyway. The vector record now carries the keys the collection declares: EventMemory's reserved timestamp, and the `_`-prefixed system properties an adapter stamps on the event, which after MemMachine#1670 are exactly the collection's schema. The vector store is queried with the conjuncts of the filter that name only such fields; a conjunct is dropped whole when any field under it is a user property, so dropping only ever widens the vector search, and the segment store narrows it back on the windows. User keys never reach the vector store, so a tenant's properties cannot shape what it stores or scans. `filter_fields` joins the filter parser: every field name a tree addresses. Rebased onto MemMachine#1631, where the vector record no longer carries the segment uuid (the segment store maps a derivative to its segment). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

xiongzubiao

The PR description says: "The squash also carries #1606's side commit regenerating docs/openapi.json under the locked FastAPI (ValidationError gains input and ctx), seven lines of that file's diff". That isn't true of this diff. Its docs/openapi.json changes are 53 deletions and no additions. input and ctx are already on feat/horizontal-scaling (docs/openapi.json:5921-5924), added by d5ec7c7 (#1733). The seven added lines in #1606's version of that file are the only part of its diff that this PR doesn't have.

I found no references to properties_schema left anywhere in the repo at this head. That covers source, tests, both OpenAPI files, the TS client, integrations, examples and the sample configs; indexed_properties_schema is a separate setting.

xiongzubiao · 2026-10-09T19:13:18Z

            ),
        ),
    ]
-    properties_schema: Annotated[


None of ProjectConfig, the memory-configuration update models, or the server config models set extra="forbid". So a properties_schema that a REST client still sends in create-project or the memory-configuration PUT is dropped silently, and the request returns 200. The same happens when an existing YAML or stored config still has the field. Python SDK callers get a TypeError for the removed keyword argument. REST callers get no sign that the field was ignored.

xiongzubiao · 2026-10-09T19:13:18Z


        def _check(field: str) -> str:
-            internal_name, is_user_metadata = normalize_filter_field(field)
+            _internal_name, is_user_metadata = normalize_filter_field(field)


Nothing reads _internal_name now; _, is_user_metadata = normalize_filter_field(field) would make that plain. Minor.

xiongzubiao

The description still says this PR carries the regeneration of docs/openapi.json from #1606, but this diff only deletes from that file. input and ctx came in with #1733. Please drop that sentence.

edwinyyyu · 2026-10-09T20:10:50Z

I think the order of this and #1702 should be reversed.

marvinyu-memverge

Approve at 5873739, on #1702's 6cdbe07. The removal is complete and nothing that depended on the per-project schema is left behind.

Verified at the head: the port commit is the same patch I read at ea69766 (git range-diff reports it unchanged), with long_term_memory.py and service_locator.py identical at both heads; no remaining use of the name in packages/ (server, common, client, ts-client), integrations/, examples/, evaluation/, tools/, docs/, design/, sample_configs/, deployments/ or the wizard; with #1702 beneath, a vector record carries only the keys the collection declares, which this PR reduces to _timestamp plus the EVENT_BACKEND_SYSTEM_FIELDS the service locator creates it with, so no caller key reaches the vector store, and a filter conjunct naming any m.<key> is dropped whole from the vector query and evaluated by the segment store, as the body says; the _-prefix reservation survives in LongTermMemory._episode_to_event, so system fields still cannot be spoofed through filterable_metadata; _validate_event_backend_filter checks bare names only and passes any m.<key>; the session collection open-or-create path is unchanged apart from the dropped spread; PROPERTY_TYPE_NAME_TO_PROPERTY_TYPE still has users (properties_json.py, vector_store/data_types.py). This also closes the REST case from my #1733 ask 1 for collections created from here on. CI: unit tests, ty, ruff, docs and the client package green; the eight red integration jobs are Docker Hub's unauthenticated pull limit (toomanyrequests, no image pulled, no test ran), the same failure #1702 shows at the same commit.

PRs have been reordered. The correctness need in #1776 has been fixed by #1813. This is for performance/this changes the contracts more, so it deserves more review based on the final state as in #1814.

The event backend wrote every property of an event into its vector record and mapped the caller's whole filter onto the vector store, so a user key that a deployment never declared was both stored and filtered there: on Qdrant and Milvus as unindexed payload a filtered query scans for. The segment store already holds every property and already receives the whole filter for the context windows, so the vector side only duplicated work the segment store does anyway. The vector record now carries the keys the collection declares: EventMemory's reserved timestamp, the `_`-prefixed system properties an adapter stamps on the event, and the keys a project's `properties_schema` declares. The vector store is queried with the conjuncts of the filter that name only such fields; a conjunct is dropped whole when any field under it is undeclared, so dropping only ever widens the vector search, and the segment store narrows it back on the windows. An undeclared key never reaches the vector store, so a tenant's undeclared properties cannot shape what it stores or scans. `filter_fields` joins the filter parser: every field name a tree addresses. Rebased onto MemMachine#1631, where the vector record no longer carries the segment uuid (the segment store maps a derivative to its segment). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…ck) (MemMachine#1606) * Regenerate the OpenAPI document under the locked FastAPI `docs/openapi.json` predates the FastAPI release in `uv.lock` (0.141.1), whose `ValidationError` component carries `input` and `ctx`; regenerating the document with `docs/tools/generate_openapi.py` adds the two fields and changes nothing else. Separate from the API changes above it so their diffs of this file show only what they change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn * Remove per-project filterable properties A project could declare `properties_schema`, a set of caller property keys with types, on its long-term memory configuration; the event backend merged it into the vector store collection's indexed schema and rejected filters on any other `m.<key>`. That let a tenant create database resources (indexes, columns) by naming them in a request, which is what forced per-collection native resources named by a hash of their schema on the backends that limit them. The option is removed from the server configuration, the project API and the memory-configuration API, the Python SDK, the sample configurations, the configuration docs and the OpenAPI document. A filter may name any `m.<key>`; the stores evaluate it on the properties they hold. What a store indexes is decided by the deployment, not per project. A breaking API change on `speedkick`. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Rebased onto MemMachine#1631: the per-project schema also leaves MemMachine#1631's service locator, which creates the session's collection in a retry loop, and the commented option goes from the event sample configuration MemMachine#1698 added. --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]> (cherry picked from commit a8322a7)

edwinyyyu force-pushed the port/remove-per-project-filterable-properties-main branch from 71f3100 to 548494e Compare September 17, 2026 17:39

edwinyyyu changed the title ~~[speedkick port 16/17] Remove per-project filterable properties (port of #1606)~~ [vector store 4/16] Remove per-project filterable properties (port of #1606) Sep 17, 2026

This was referenced Sep 17, 2026

[sqlite store fixes 3/7] Own the search engine's concurrency in the store, not in each engine (port of #1612) #1672

Draft

[sqlite store fixes 4/7] Serialize a partition's writes so the engine sees them in order (port of #1607) #1673

Draft

This was referenced Oct 7, 2026

Classify Qdrant errors by status code alone #1779

Merged

Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract #1788

Open

edwinyyyu force-pushed the port/remove-per-project-filterable-properties-main branch from 0e09327 to 4bc79e8 Compare October 7, 2026 21:16

edwinyyyu mentioned this pull request Oct 7, 2026

Refuse property values that some store refuses or alters where an episode enters #1792

Draft

edwinyyyu force-pushed the port/remove-per-project-filterable-properties-main branch 3 times, most recently from d156084 to b65961f Compare October 9, 2026 00:52

edwinyyyu changed the base branch from main to feat/horizontal-scaling October 9, 2026 00:52

edwinyyyu force-pushed the port/remove-per-project-filterable-properties-main branch from b65961f to ea69766 Compare October 9, 2026 17:53

edwinyyyu requested review from malatewang, marvinyu-memverge and xiongzubiao October 9, 2026 17:56

edwinyyyu marked this pull request as ready for review October 9, 2026 17:56

xiongzubiao reviewed Oct 9, 2026

View reviewed changes

xiongzubiao previously approved these changes Oct 9, 2026

View reviewed changes

edwinyyyu marked this pull request as draft October 9, 2026 20:10

edwinyyyu force-pushed the port/remove-per-project-filterable-properties-main branch from ea69766 to 75a0d21 Compare October 9, 2026 20:12

edwinyyyu changed the title ~~[user properties 1/2] Remove per-project filterable properties (port of #1606)~~ [user properties 2/2] Remove per-project filterable properties (port of #1606) Oct 9, 2026

edwinyyyu mentioned this pull request Oct 9, 2026

The server's "session" is a project's whole memory scope, but nothing defines it and the name reads as a conversation #1812

Open

edwinyyyu force-pushed the port/remove-per-project-filterable-properties-main branch from 75a0d21 to 5873739 Compare October 9, 2026 20:22

edwinyyyu marked this pull request as ready for review October 9, 2026 21:02

marvinyu-memverge approved these changes Oct 9, 2026

View reviewed changes

This was referenced Oct 9, 2026

[final state] The vector store tree combined, for reference (not for merge) #1814

Draft

[final state, core] The vector store tree's contract changes combined, for reference (not for merge) #1815

Draft

edwinyyyu and others added 2 commits October 9, 2026 17:04

edwinyyyu force-pushed the port/remove-per-project-filterable-properties-main branch from 5873739 to 7992829 Compare October 10, 2026 00:32

Navigation Menu

Repository navigation

[user properties 2/2] Remove per-project filterable properties (port of #1606) - #1670

[user properties 2/2] Remove per-project filterable properties (port of #1606)#1670
edwinyyyu wants to merge 2 commits into
MemMachine:feat/horizontal-scalingfrom
edwinyyyu:port/remove-per-project-filterable-properties-main

edwinyyyu commented Sep 16, 2026 •

edited

Loading

Uh oh!

xiongzubiao left a comment

Uh oh!

xiongzubiao Oct 9, 2026

Uh oh!

xiongzubiao Oct 9, 2026

Uh oh!

xiongzubiao left a comment

Uh oh!

edwinyyyu commented Oct 9, 2026

Uh oh!

marvinyu-memverge left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

edwinyyyu commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Port

Purpose of the change

Stack

Verification

Uh oh!

xiongzubiao left a comment

Choose a reason for hiding this comment

Uh oh!

xiongzubiao Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

xiongzubiao Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

xiongzubiao left a comment

Choose a reason for hiding this comment

Uh oh!

edwinyyyu commented Oct 9, 2026

Uh oh!

marvinyu-memverge left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

edwinyyyu commented Sep 16, 2026 •

edited

Loading