Skip to content

Make no memory request create a project - #1624

Draft
edwinyyyu wants to merge 3 commits into
MemMachine:mainfrom
edwinyyyu:feat/no-implicit-project-speedkick
Draft

edwinyyyu wants to merge 3 commits into
MemMachine:mainfrom
edwinyyyu:feat/no-implicit-project-speedkick

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the change

The simple chatbot example, the TypeScript REST demo and the Dify plugin's add-memory tool wrote to a project without creating it, relying on the write to create it. Each now creates its project before its first memory request and accepts 409 as the project already existing. No behavior changes for them; they stop depending on a write creating a project.

Adding memories to, or searching, a project that did not exist created it, with the server's default configuration, without the caller's knowledge. Now only the create-project request creates a project: a write or a search opens the session and answers 404 for an unknown project, as the search endpoint already promised; the manager's open-or-create goes.

Two callers depended on the implicit creation. org_id and project_id default to universal, so the API promises the project universal/universal; the server creates it, once, at startup, and leaves one that already exists as it is. The MCP add tool names its own project and has no create-project counterpart, so it creates the project it writes to, once, and says so. The API doc strings and the OpenAPI document say which requests create a project.

A breaking API change on main.

Adaptation for main: add_memories keeps main's default of all memory types (#1555 is not ported); on main a session's storage is still created lazily by the first open, which #1622 in the chain moves to creation. The deleted-session test keeps only its open assertion: on speedkick it also pinned that re-creating a session being deleted is refused, and #1677's create_or_validate_session accepts one whose data matches (#1539, which it ports, refused it); reported on #1677 rather than fixed here.

Stack

21 open PRs: three independent PRs, and the vector store tree of short parallel branches. Every PR in the tree has feat/horizontal-scaling as its GitHub base, and the independent PRs have main. The branches are in a fork, and a pull request can target only this repository's branches, so the on column gives the order the PRs build on each other. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

# PR change on
— #1624 (this PR) Make no memory request create a project main
— #1786 Refuse a filter value of the wrong type for a datetime column, and answer an invalid list filter with 422 (port of #1620) main
— #1792 Refuse property values that some store refuses or alters where an episode enters main

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1702 and #1670 sit beneath #1627, whose code depends on them. Until the PRs under it merge, their changes show in a stacked PR's diff.

# PR change on
[vector store scale-out 1/6] #1671 (merged) Remove custom sharding from the Qdrant store (port of #1654) main
[vector store scale-out 2/6] #1733 (merged into feat/horizontal-scaling) Answer vector store queries with record UUIDs and scores, and refuse invalid inputs feat/horizontal-scaling
[vector store scale-out 3/6] #1734 (merged into feat/horizontal-scaling) Arbitrate vector store collections in a SQL registry, with an incarnation per collection life and a purge feat/horizontal-scaling
[vector store scale-out 4/6] #1735 (merged into feat/horizontal-scaling) Move the Qdrant store onto the collection registry feat/horizontal-scaling
[vector store scale-out 5/6] #1736 (merged into feat/horizontal-scaling) Move the Milvus store onto the collection registry, against a Milvus server feat/horizontal-scaling
— #1775 (merged into feat/horizontal-scaling) Accept attempts exhausted in the lifecycle churn contract, and say which Qdrant operations filter on the incarnation feat/horizontal-scaling
— #1779 (merged into feat/horizontal-scaling) Classify Qdrant errors by status code alone feat/horizontal-scaling
— #1813 (merged into feat/horizontal-scaling) Create Qdrant collections with strict mode off feat/horizontal-scaling
— #1788 Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract feat/horizontal-scaling
— #1631 (closed) Superseded by #1733–#1736, which hold its changes split in four, with review changes since —
[user properties 1/2] #1702 Keep undeclared properties out of the vector store feat/horizontal-scaling
[user properties 2/2] #1670 Remove per-project filterable properties (port of #1606) #1702
[vector store scale-out 6/6] #1627 Make a vector store one collection, with string-keyed partitions #1670
[session storage 1/2] #1622 Create a session's storage with the session, never on a request #1627
[session storage 2/2] #1625 Remove open-or-create from both stores, and close from the segment store #1622
[declared schema 1/2] #1628 Make a vector store filter only on the properties it declares #1627
[search results] #1663 Score every vector search by cosine similarity, and name scores for it (port of #1598's cosine half) #1628
[declared schema 2/2] #1616 Close the filter union, and make negation the complement on every backend #1663
[sqlite store fixes 1/7] #1460 Publish vector index files atomically (but not durably) #1663
[sqlite store fixes 2/7] #1469 Never reuse a row id in SQLiteVectorStore #1460
[sqlite store fixes 3/7] #1672 Own the search engine's concurrency in the store, not in each engine (port of #1612) #1469
[sqlite store fixes 4/7] #1673 Serialize a partition's writes so the engine sees them in order (port of #1607) #1672
[sqlite store fixes 5/7] #1674 Refuse a pending row replay cannot honor, instead of dropping it (port of #1608) #1673
[sqlite store fixes 6/7] #1675 Take SQLite's write lock at BEGIN, not at the first write (port of #1609) #1674
[sqlite store fixes 7/7] #1676 Give every write a fresh row id, so a key names one version (port of #1610) #1675
[qdrant options] #1618 Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization #1663
[milvus options] #1741 Let a deployment tune a Milvus collection's vector index and its searches #1618

This PR is its 3 commits, 555ef9cde, 17f12e541, 5efe2a3a2, directly on main, independent of the vector store tree: review and merge it in any order.

Verification

At every one of its 3 commits, on 2026-09-21: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); the full server suite without integration tests passes (pytest packages/server/server_tests -m "not integration"), 1921 tests at its head 5efe2a3a2.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

@edwinyyyu
edwinyyyu force-pushed the feat/no-implicit-project-speedkick branch 5 times, most recently from 15ee80d to 1639f5e Compare September 14, 2026 23:16
@edwinyyyu edwinyyyu changed the title [vector store 5/13] Make no memory request create a project (speedkick) [vector store 5/12] Make no memory request create a project (speedkick) Sep 14, 2026
@edwinyyyu
edwinyyyu force-pushed the feat/no-implicit-project-speedkick branch 2 times, most recently from 15ee80d to e1075ec Compare September 15, 2026 17:25
@edwinyyyu edwinyyyu changed the title [vector store 5/12] Make no memory request create a project (speedkick) [vector store 5/13] Make no memory request create a project (speedkick) Sep 15, 2026
@edwinyyyu
edwinyyyu force-pushed the feat/no-implicit-project-speedkick branch 5 times, most recently from a1025f9 to 9b3c734 Compare September 15, 2026 19:16
@edwinyyyu
edwinyyyu force-pushed the feat/no-implicit-project-speedkick branch from 9b3c734 to a8c4b22 Compare September 15, 2026 19:47
@edwinyyyu
edwinyyyu force-pushed the feat/no-implicit-project-speedkick branch from d21a5d0 to ffcdd2c Compare September 17, 2026 19:32
@edwinyyyu edwinyyyu changed the title [vector store 8/16] Make no memory request create a project Make no memory request create a project Sep 17, 2026

@edwinyyyu edwinyyyu left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. The no-implicit-project guarantee has a gap for non-episodic requests (line comment).
  2. memmachine-server --stdio (app.py) calls initialize_resource() directly and never init_global_memory, so ensure_default_project does not run for that entrypoint; "at startup" covers the HTTP lifespan, mcp_http and mcp_stdio only. Same on the closed speedkick counterpart, #1639.

packages/server/src/memmachine_server/main/memmachine.py:710 -- The 404 for an unknown project is raised only inside if MemoryType.Episodic in target_memories. A write with types: ["semantic"] (any request without episodic) to an unknown project is answered 200: episode_storage.add_episodes a few lines up has already inserted rows under the unknown session key (no FK to the sessions table), and the semantic manager ingests without a session lookup. Even with episodic included, the episode rows are persisted before this check runs, so the 404 arrives after the write. The description's "a write or a search opens the session and answers 404 for an unknown project" holds only for episodic requests. Same on #1639.

@edwinyyyu edwinyyyu mentioned this pull request Sep 18, 2026
4 of 26 tasks
@edwinyyyu
edwinyyyu force-pushed the feat/no-implicit-project-speedkick branch from e1b6052 to 4761a70 Compare September 18, 2026 19:39
edwinyyyu and others added 3 commits September 21, 2026 11:09
The simple chatbot example, the TypeScript REST demo and the Dify plugin's
add-memory tool wrote to a project without creating it, relying on the
write to create it. Each now creates its project before its first memory
request and accepts 409 as the project already existing. No behavior
changes for them; they stop depending on a write creating a project.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
(cherry picked from commit 4cae58a)
Adding memories to, or searching, a project that did not exist created it,
with the server's default configuration, without the caller's knowledge.
Now only the create-project request creates a project: a write or a
search opens the session and answers 404 for an unknown project, as the
search endpoint already promised; the manager's open-or-create goes.

Two callers depended on the implicit creation. `org_id` and `project_id`
default to `universal`, so the API promises the project
`universal/universal`; the server creates it, once, at startup, and leaves
one that already exists as it is. The MCP add tool names its own project
and has no create-project counterpart, so it creates the project it writes
to, once, and says so. The API doc strings and the OpenAPI document say
which requests create a project.

A breaking API change on `speedkick`.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
(cherry picked from commit d21a5d0)
…anything, on every entry point

A write persisted its episodes before opening episodic memory, and a
write or search that targeted only semantic memory never opened it, so a
request naming a project nobody created could still insert episode rows
under that key and answer 200. Every write now checks the registry first,
and a search that does not open episodic memory checks it too; the check
is one registry read, and the episodic open, which refuses an unknown
project itself, is unchanged.

`memmachine-server --stdio` built its resources without setting or
starting the module-level MemMachine the tools read, and so never created
the default project either; it now starts and stops through the same
calls as the HTTP servers.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu
edwinyyyu force-pushed the feat/no-implicit-project-speedkick branch from 4761a70 to 5efe2a3 Compare September 21, 2026 18:12
@edwinyyyu edwinyyyu added horizontal scaling Wrong or unsafe when more than one server process serves the same backends (replicas or workers) underspecified behavior and removed horizontal scaling Wrong or unsafe when more than one server process serves the same backends (replicas or workers) labels Sep 29, 2026
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Oct 10, 2026
…uilt by the composition root

A vector store was a factory of logical collections, each identified by
a (namespace, name) pair and created with its own dimensions, metric and
schema; Qdrant and Milvus shared one native collection among logical
collections of equal configuration, under a name derived from a hash of
that configuration, and the registry mapped (namespace, name) to it.

A store is now one collection: `VectorStore(vector_store_name,
vector_dimensions, similarity_metric, indexed_properties)` names its one
native collection (or its tables and index files) at construction, every
partition of it shares the store's dimensions, metric and schema, and
`provision()` creates the store's durable resources idempotently, before
`startup`. `create_partition(key)`, `open_or_create_partition(key)`,
`get_partition(key)` and `delete_partition(key)` take a string key; a
partition is the records carrying its incarnation in the store's native
collection (Qdrant, Milvus) or a pair of tables (the SQLite stores), and
the registry records what each partition was created under, so a store
built with other dimensions, another metric or another schema raises
`VectorStorePartitionSchemaMismatchError` instead of reading columns and
vectors that are not there. "Collection" names only the native Qdrant or
Milvus collection. Vector store names match `[a-z0-9_]+` and are at most
32 bytes, the rule partition and property keys follow, so every name
works on every backend: the Qdrant store names its collection by the
vector store name, and the Milvus store by `sys_` followed by it, since
Milvus requires a leading letter or underscore and names its own
internals with a leading underscore. The hash-derived native names go,
and with them the per-partition config.

The partition registry is keyed by partition key within one store: its
tables, `partition_registry_pt` and `partition_registry_gc`, are shared
by every store on a database and keyed by vector store name, so the
registries of several stores share one database and nothing else.
`mark_live` and `unregister_incarnation` take the incarnation alone. On
the registry-backed base, the storage a store's partitions share is
prepared by `provision()` (`_prepare_storage()`), and the storage a
partition keeps of its own by `_prepare_partition_storage(key,
incarnation)`, between its registration and its mark; the Qdrant and
Milvus stores keep nothing per partition. A purge round works on one
incarnation in the store's own collection.

`DatabaseManager.get_vector_store(backend, vector_store_name=,
vector_dimensions=, similarity_metric=, indexed_properties=)` builds and
caches one store per (backend, vector store name), provisioning its
registry and then the store; asking for a name again with other
dimensions, another metric or other keys is a
`VectorStoreConfigurationError`. The event backend's store is named by a
UUIDv5 of its embedder id in the UUIDv5 of its backend's key in a namespace
fixed for the event backend, written as 32 hexadecimal digits, so neither
the key nor the id is constrained; stores on different backends never share
a name, so their registries stay apart when the backends share a registry
database, since a store name identifies one store among all the stores
whose registries share it.
Semantic memory's one store is named `semantic_memory` whatever its
embedder and holds every org's features in one partition,
`semantic_memory`, as main's one collection does.
The event backend opens a session's partition with
`open_or_create_partition`, which waits on one another worker is still
creating.

The SQLite stores change shape only: their registry tables become
`vector_store_sqlite_pt` and `vector_store_sqlite_vec_pt`, keyed by
vector store name and partition key, the pending-operation log is keyed
the same way, and per-partition table names embed the vector store name.
No migration is provided; existing SQLite vector data is orphaned.

Rebuilt as one change on MemMachine#1631: this PR's earlier history carried copies
of MemMachine#1631's commits as of `38684b385`, its own commits on them, and
mirrors of MemMachine#1631's later commits in its terms. Its code is that history's
final tree, which was verified, merged with main and without MemMachine#1624's
commits. The design documents are still MemMachine#1631's and describe collections.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant