Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract by edwinyyyu · Pull Request #1788 · MemMachine/MemMachine

edwinyyyu · 2026-10-07T18:31:01Z

Purpose of the change

Summary

Every vector store (SQLiteVectorStore, SQLiteVecVectorStore, and the registry-backed handle behind QdrantVectorStore and MilvusVectorStore) refuses an upsert batch that names a record UUID twice: it raises ValueError naming the UUID before anything is sent to the backend. Two versions of one record in a batch are ambiguous, and the stores disagreed on them: SQLiteVectorStore and Qdrant kept the last, SQLiteVecVectorStore failed partway with a raw sqlite3 UNIQUE constraint error, and Milvus refused the batch. The check costs one pass over the batch.
Every store refuses a non-finite float property value (nan, inf, or -inf), declared or undeclared: it raises ValueError naming the property key before anything is sent, as the query path already refuses a non-finite query vector coordinate or score threshold. Qdrant over REST stored such a value as null and over gRPC refused it with a raw error, Milvus refused it in a declared float field, took it in an undeclared one, and cannot parse a filter on it, and the SQLite stores stored it.
The VectorStoreCollection docstring states the datetime property contract: a filter compares datetime values by their instant, for equality and ordering alike, and a store keeps a datetime property's instant, taking a naive datetime as UTC, and may drop its UTC offset. A search answers with record UUIDs and scores, so no caller reads a stored offset back; the segment store the vector records are derived from keeps it. The tests read the instant back from each backend and filter on it at other offsets.
Qdrant now writes a datetime property as its UTC instant. A datetime written to Qdrant keeps its offset only to the minute, over REST and gRPC alike (on 1.19.1, one written at -00:44:30 reads back at -00:44), so one at an offset with a seconds component was stored up to 59 seconds off its instant. Converting to UTC first applies the whole offset to the instant.

Both checks are helpers in common/vector_store/utils.py, beside the other upsert input checks, and the VectorStoreCollection.upsert docstring lists them under Raises.

Directly on feat/horizontal-scaling, where #1736 is merged.

Fixes #1783. Fixes #1784.

Commits

Refuse an upsert batch that names a record UUID twice
Refuse a non-finite float property value at upsert
State the datetime property contract of a vector store collection
Promise a datetime property's instant, not its offset
Write a Qdrant datetime as its UTC instant

Stack

21 open PRs: three independent PRs, and the vector store tree of short parallel branches. Every PR in the tree has feat/horizontal-scaling as its GitHub base, and the independent PRs have main. The branches are in a fork, and a pull request can target only this repository's branches, so the on column gives the order the PRs build on each other. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

#	PR	change	on
—	#1624	Make no memory request create a project	`main`
—	#1786	Refuse a filter value of the wrong type for a datetime column, and answer an invalid list filter with 422 (port of #1620)	`main`
—	#1792	Refuse property values that some store refuses or alters where an episode enters	`main`

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1702 and #1670 sit beneath #1627, whose code depends on them. Until the PRs under it merge, their changes show in a stacked PR's diff.

#	PR	change	on
[vector store scale-out 1/6]	#1671 (merged)	Remove custom sharding from the Qdrant store (port of #1654)	`main`
[vector store scale-out 2/6]	#1733 (merged into `feat/horizontal-scaling`)	Answer vector store queries with record UUIDs and scores, and refuse invalid inputs	`feat/horizontal-scaling`
[vector store scale-out 3/6]	#1734 (merged into `feat/horizontal-scaling`)	Arbitrate vector store collections in a SQL registry, with an incarnation per collection life and a purge	`feat/horizontal-scaling`
[vector store scale-out 4/6]	#1735 (merged into `feat/horizontal-scaling`)	Move the Qdrant store onto the collection registry	`feat/horizontal-scaling`
[vector store scale-out 5/6]	#1736 (merged into `feat/horizontal-scaling`)	Move the Milvus store onto the collection registry, against a Milvus server	`feat/horizontal-scaling`
—	#1775 (merged into `feat/horizontal-scaling`)	Accept attempts exhausted in the lifecycle churn contract, and say which Qdrant operations filter on the incarnation	`feat/horizontal-scaling`
—	#1779 (merged into `feat/horizontal-scaling`)	Classify Qdrant errors by status code alone	`feat/horizontal-scaling`
—	#1813 (merged into `feat/horizontal-scaling`)	Create Qdrant collections with strict mode off	`feat/horizontal-scaling`
—	#1788 (this PR)	Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract	`feat/horizontal-scaling`
—	#1631 (closed)	Superseded by #1733–#1736, which hold its changes split in four, with review changes since	—
[user properties 1/2]	#1702	Keep undeclared properties out of the vector store	`feat/horizontal-scaling`
[user properties 2/2]	#1670	Remove per-project filterable properties (port of #1606)	#1702
[vector store scale-out 6/6]	#1627	Make a vector store one collection, with string-keyed partitions	#1670
[session storage 1/2]	#1622	Create a session's storage with the session, never on a request	#1627
[session storage 2/2]	#1625	Remove open-or-create from both stores, and close from the segment store	#1622
[declared schema 1/2]	#1628	Make a vector store filter only on the properties it declares	#1627
[search results]	#1663	Score every vector search by cosine similarity, and name scores for it (port of #1598's cosine half)	#1628
[declared schema 2/2]	#1616	Close the filter union, and make negation the complement on every backend	#1663
[sqlite store fixes 1/7]	#1460	Publish vector index files atomically (but not durably)	#1663
[sqlite store fixes 2/7]	#1469	Never reuse a row id in SQLiteVectorStore	#1460
[sqlite store fixes 3/7]	#1672	Own the search engine's concurrency in the store, not in each engine (port of #1612)	#1469
[sqlite store fixes 4/7]	#1673	Serialize a partition's writes so the engine sees them in order (port of #1607)	#1672
[sqlite store fixes 5/7]	#1674	Refuse a pending row replay cannot honor, instead of dropping it (port of #1608)	#1673
[sqlite store fixes 6/7]	#1675	Take SQLite's write lock at BEGIN, not at the first write (port of #1609)	#1674
[sqlite store fixes 7/7]	#1676	Give every write a fresh row id, so a key names one version (port of #1610)	#1675
[qdrant options]	#1618	Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization	#1663
[milvus options]	#1741	Let a deployment tune a Milvus collection's vector index and its searches	#1618

This PR is its 5 commits, 299d9d41a, 779cc8d41, 299e5902e, e136eb082, 18e89e062, directly on feat/horizontal-scaling.

Verification

Rebased onto feat/horizontal-scaling cd96676bc, #1736's squash merge: the 5 commits replay unchanged, and the head's tree differs from 3fe0b442b only by #1779's change to qdrant_vector_store.py, which the merge brought. At the head 18e89e062, in order:

uv run --frozen --all-extras ruff check: all checks passed.
uv run --frozen --all-extras ruff format --check: 553 files already formatted.
uv run --frozen --all-extras ty check --project packages/server: all checks passed.
uv lock --check: passed.
uv run --frozen --all-extras pytest packages/server -m "not integration" -q: 2,189 passed, 8 skipped.
uv run --frozen --all-extras pytest packages/server -m integration -q (Qdrant 1.19.1, Milvus 2.6.24, and PostgreSQL in testcontainers): 1,712 passed, 266 skipped.

Rebased onto #1736's head 49b9ec2dd, with commits 4 and 5: at the head 3fe0b442b, in order:

uv run --frozen --all-extras ruff check: all checks passed.
uv run --frozen --all-extras ruff format --check: 553 files already formatted.
uv run --frozen --all-extras ty check --project packages/server: all checks passed.
uv lock --check: passed.
uv run --frozen --all-extras pytest packages/server -m "not integration" -q: 2,189 passed, 8 skipped.
uv run --frozen --all-extras pytest packages/server -m integration -q (Qdrant 1.19.1, Milvus 2.6.24, and PostgreSQL in testcontainers): 1,712 passed, 266 skipped.

Commits 1-3 replay unchanged. Commit 5's tests write a datetime at -00:44:30, under a declared key and an undeclared one, over REST and gRPC, read the payload back in UTC, and filter on its instant with =, <, and >; without commit 5 all 8 cases fail.

Before the rebase:

At 61c2045, in order:

uv run --frozen --all-extras ruff check: all checks passed.
uv run --frozen --all-extras ruff format --check: 553 files already formatted.
uv run --frozen --all-extras ty check --project packages/server: all checks passed.
uv lock --check: passed.
uv run --frozen --all-extras pytest packages/server -m "not integration" -q: 2,180 passed, 8 skipped.
uv run --frozen --all-extras pytest packages/server -m integration -q (Qdrant 1.19.1, Milvus 2.6.24, and PostgreSQL in testcontainers): 1,704 passed, 266 skipped.

Each new or changed test failed under its named mutation, on every store it runs on:

The repeated-UUID tests, with the check removed: SQLiteVectorStore and Qdrant (REST and gRPC) raised nothing, SQLiteVecVectorStore raised sqlite3's OperationalError, and Milvus raised MilvusException (code 1100).
The non-finite property tests, with the check removed: the SQLite stores, Qdrant over REST, and Milvus on an undeclared property raised nothing; Qdrant over gRPC raised AioRpcError (INVALID_ARGUMENT); and Milvus on a declared property raised MilvusException (code 1100).
The datetime read-back tests, with the offset dropped at write: the type-tagged JSON's offset written as 0 (the SQLite stores and Milvus's undeclared properties), Qdrant's payload converted to UTC, or Milvus's offset field written as 0.
The datetime filter tests, with the filter node reading a value's wall-clock time as UTC, on every store, and on the SQLite stores also with the wall-clock text stored in place of the UTC instant.

🤖 Generated with Claude Code

Two versions of one record in one batch are ambiguous, and the stores disagreed on them: SQLiteVectorStore and Qdrant kept the last, SQLiteVecVectorStore failed partway with a raw sqlite3 UNIQUE constraint error, and Milvus refused the batch. Every store now raises ValueError naming the UUID before anything is sent to the backend, from one helper beside the other upsert input checks; the check costs one pass over the batch. The VectorStoreCollection.upsert docstring says so, and the registry-backed _upsert hook may rely on distinct UUIDs. Each store's test upserts a batch repeating a UUID after another record, expects the error, and finds nothing stored. It fails with the check removed: SQLiteVectorStore and Qdrant raise nothing, SQLiteVecVectorStore raises sqlite3's OperationalError, and Milvus 2.6.24 raises its MilvusException. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

The stores disagreed on nan, inf, and -inf as property values: Qdrant over REST stores them as null, so the property reads back absent, and over gRPC refuses them with a raw error; Milvus 2.6.24 refuses one in a declared float field, takes one in an undeclared property, and cannot parse a filter on it; and the SQLite stores store them. Every store now raises ValueError naming the property key before anything is sent to the backend, for declared and undeclared properties alike, as the query path already refuses a non-finite query vector coordinate or score threshold. The VectorStoreCollection.upsert docstring says so, and the registry-backed _upsert hook may rely on finite values. Each store's test upserts a finite record and one with a non-finite value of each kind, on a declared and an undeclared property, expects the error, and finds nothing stored. It fails with the check removed: the SQLite stores, Qdrant over REST, and Milvus on an undeclared property raise nothing; Qdrant over gRPC raises grpc's AioRpcError; and Milvus on a declared property raises its MilvusException. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

A filter compares datetime values by their instant, for equality and ordering alike, and a store keeps a datetime property's UTC offset as written, taking a naive datetime as UTC. Every store behaves so: the SQLite stores keep the offset beside the UTC instant in their type-tagged JSON, Qdrant keeps the RFC 3339 text with its offset in the payload, and Milvus keeps a declared datetime's offset in a field beside its TIMESTAMPTZ and an undeclared one's in its properties JSON. The VectorStoreCollection docstring says so. Each store's tests write a datetime at +05:30, declared and undeclared, read its offset back from the backend, and filter on it at other offsets: the same instant at -08:00 matches =, <=, and >=, a later instant at an earlier wall-clock time matches <, and an earlier instant at a later wall-clock time matches >. The read-back tests fail with the offset dropped at write (the type-tagged JSON's offset written as 0, Qdrant's payload converted to UTC, or Milvus's offset field written as 0). The filter tests fail with the filter node reading a value's wall-clock time as UTC, and on the SQLite stores also with the wall-clock text stored in place of the UTC instant. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

A vector store's properties exist for filters, which compare a datetime by its instant, and a search answers with record UUIDs and scores, so no caller reads a stored offset back. The VectorStoreCollection docstring now says a store keeps a datetime property's instant, taking a naive datetime as UTC, and may drop its UTC offset; a filter compares datetime values by their instant, as before. Qdrant's read-back test checks the stored instant, not its offset. The tests of the type-tagged JSON, which keeps the offset (the SQLite stores and Milvus's undeclared properties), are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

A datetime written to Qdrant keeps its UTC offset only to the minute, over REST and gRPC alike: on 1.19.1, one written at -00:44:30 reads back at -00:44. The store wrote a datetime property at its own offset, so one at an offset with a seconds component, as local mean time had until 1972, was stored up to 59 seconds off its instant, and a filter on that instant missed it: the store did not keep the instant the collection's contract promises. The store now converts the datetime to UTC before writing it, which applies the whole offset to the instant, so nothing is left for Qdrant to cut. The read-back test checks that the payload holds the written instant in UTC. The new filter test writes a datetime at -00:44:30, under a declared key and an undeclared one, over REST and gRPC, and filters on its instant with =, <, and >. Without the change, all 8 of their cases fail. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

marvinyu-memverge

Approving at 18e89e0. No asks.

What I checked:

Both new checks run before anything is sent, on all four stores. The registry-backed handle (Qdrant, Milvus) and both SQLite stores check every record's properties, then the batch's UUIDs, ahead of the first write. So "no record is written" in the upsert docstring holds.
No caller today sends a repeated UUID in one batch. Event memory's derivatives each get a fresh uuid4 in the text deriver, and semantic memory upserts one record at a time with its own vector_uuid. The UUID check refuses nothing that is produced now; it pins the contract the stores disagreed on.
The filter side already agrees with the new Qdrant write. The filter parser's Comparison and In nodes convert every datetime value to UTC when they are built, so Qdrant's eq and range filters never send a seconds offset. That is why the -00:44:30 =, < and > cases hold over both transports.
The SQLite stores (encode_properties) and Milvus keep the UTC instant plus the original offset, so Qdrant is the only store that drops the offset. That matches "may drop its UTC offset" in the VectorStoreCollection docstring and the new line in design/qdrant_vector_store.md.
CI is green at head on every job, including the common/ integration matrix.

A declared datetime on the SQLite stores had a `tz_<key>` column beside its microseconds column, holding the UTC offset in seconds that no filter reads. The vector stores keep a datetime property's instant only, as MemMachine#1736 now does for Milvus and MemMachine#1788 for Qdrant, and the segment store keeps the offset, so the column, `offset_column_name`, and the value written to it go: each declared property is one column, and a datetime stays microseconds since the epoch. The tests read a stored datetime back as its instant in UTC, and the roundtrip test checks that the stored instant equals the written one. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Every store refuses a batch naming a record UUID twice. A non-finite float is already refused by the declared schema's type check, so its separate check and tests go, and the datetime tests run on the declared key alone, since an undeclared key is refused. The store keeps the strict mode of MemMachine#1628 over the one MemMachine#1813 merged into the base, and MemMachine#1813's test of it off stays out. Co-Authored-By: Claude Opus 5.5 <[email protected]>

…chine#1672 to MemMachine#1676) into the final state Their upsert keeps the delete and fresh insert, writing the declared columns, and refuses a repeated UUID as MemMachine#1788 does; open-or-create stays out, as MemMachine#1625 removes it. Their tests create their partition as a session does and build filters with the closed union of MemMachine#1616. Co-Authored-By: Claude Opus 5.5 <[email protected]>

edwinyyyu added horizontal scaling Wrong or unsafe when more than one server process serves the same backends (replicas or workers) and removed horizontal scaling Wrong or unsafe when more than one server process serves the same backends (replicas or workers) labels Oct 7, 2026

edwinyyyu force-pushed the feat/vector-store-write-validation-speedkick branch from 61c2045 to 3fe0b44 Compare October 8, 2026 23:54

edwinyyyu and others added 5 commits October 9, 2026 10:50

edwinyyyu force-pushed the feat/vector-store-write-validation-speedkick branch from 3fe0b44 to 18e89e0 Compare October 9, 2026 18:02

edwinyyyu requested review from malatewang and marvinyu-memverge October 9, 2026 22:28

marvinyu-memverge approved these changes Oct 9, 2026

View reviewed changes

Navigation Menu

Repository navigation

Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract - #1788

Refuse a repeated record UUID or a non-finite property value at upsert, and state the datetime property contract#1788
edwinyyyu wants to merge 5 commits into
MemMachine:feat/horizontal-scalingfrom
edwinyyyu:feat/vector-store-write-validation-speedkick

edwinyyyu commented Oct 7, 2026 •

edited

Loading

Uh oh!

marvinyu-memverge left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

edwinyyyu commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the change

Summary

Commits

Stack

Verification

Uh oh!

marvinyu-memverge left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

edwinyyyu commented Oct 7, 2026 •

edited

Loading