What happens
Measured on main at ad8ff24, one value at a time through each store's own code, on SQLite and PostgreSQL where a store supports both, with Qdrant 1.17.0 (REST and gRPC) and Milvus 2.6.24 and Milvus Lite. The value is a string that becomes a property value: a metadata key or value, producer, produced_for, or role of a message, which are producer_id, producer_role, produced_for_id, and metadata of an EpisodeEntry.
| store |
"a\u0000b" |
"a\ud800b" (lone surrogate) |
| episode store, SQLite |
stored |
metadata value stored; producer_id refused (UnicodeEncodeError) |
| episode store, PostgreSQL |
refused (UntranslatableCharacterError for the metadata JSONB, CharacterNotInRepertoireError for producer_id) |
refused |
| segment store, SQLite |
stored |
stored |
| segment store, PostgreSQL |
refused (UntranslatableCharacterError) |
refused (InvalidTextRepresentationError) |
| semantic memory set ids, SQLite |
stored |
refused (UnicodeEncodeError) |
| semantic memory set ids, PostgreSQL (pgvector and vector-store storage) |
refused (CharacterNotInRepertoireError) |
refused (DataError) |
SQLiteVectorStore, SQLiteVecVectorStore |
stored |
stored; an equality filter on it raises UnicodeEncodeError |
QdrantVectorStore, REST and gRPC |
stored |
refused (PydanticSerializationError, UnicodeEncodeError) |
MilvusVectorStore, 2.6.24 and Lite |
stored; on Lite an equality filter on it raises MilvusException |
refused (DataNotMatchException) |
Semantic memory receives the value inside a set id, for producer and for a metadata value whose key a set type names.
The request accepts both: JSON "\u0000" decodes to U+0000, and the route's JSON decoding turns "\ud800" into a lone surrogate. Through POST /api/v2/memories with the real stores (episode store, segment store, and semantic storage on PostgreSQL, and SQLiteVectorStore):
- metadata
{"k": "a\u0000b"} or {"k": "a\ud800b"}: 500, with the episode recorded in semantic memory's three sets and in no other store;
producer "a\u0000b": 500, nothing written;
- with the same stores on SQLite, all three: 200, stored everywhere.
Why
MemoryMessage types these fields as str (packages/common/src/memmachine_common/api/spec.py:458-477), and EpisodeEntry does too (packages/server/src/memmachine_server/common/episode_store/episode_model.py:29-35).
MemMachine.add_episodes writes the episode store, episodic memory, and semantic memory concurrently and raises the first failure once all of them return (packages/server/src/memmachine_server/main/memmachine.py:795-819), so the stores that keep the value have written it.
- PostgreSQL text and JSONB refuse U+0000, and UTF-8 encoding refuses a lone surrogate.
Expected
The request refuses such a string with 422 before any store writes, and building an EpisodeEntry with one raises.
Fix
#1792 types these fields as PropertyStr, which refuses a string with U+0000 or a surrogate code point.
🤖 Written by Claude Code (Claude Opus 5.5) on behalf of @edwinyyyu.
What happens
Measured on
mainat ad8ff24, one value at a time through each store's own code, on SQLite and PostgreSQL where a store supports both, with Qdrant 1.17.0 (REST and gRPC) and Milvus 2.6.24 and Milvus Lite. The value is a string that becomes a property value: a metadata key or value,producer,produced_for, orroleof a message, which areproducer_id,producer_role,produced_for_id, and metadata of anEpisodeEntry."a\u0000b""a\ud800b"(lone surrogate)producer_idrefused (UnicodeEncodeError)UntranslatableCharacterErrorfor the metadata JSONB,CharacterNotInRepertoireErrorforproducer_id)UntranslatableCharacterError)InvalidTextRepresentationError)UnicodeEncodeError)CharacterNotInRepertoireError)DataError)SQLiteVectorStore,SQLiteVecVectorStoreUnicodeEncodeErrorQdrantVectorStore, REST and gRPCPydanticSerializationError,UnicodeEncodeError)MilvusVectorStore, 2.6.24 and LiteMilvusExceptionDataNotMatchException)Semantic memory receives the value inside a set id, for
producerand for a metadata value whose key a set type names.The request accepts both: JSON
"\u0000"decodes to U+0000, and the route's JSON decoding turns"\ud800"into a lone surrogate. ThroughPOST /api/v2/memorieswith the real stores (episode store, segment store, and semantic storage on PostgreSQL, andSQLiteVectorStore):{"k": "a\u0000b"}or{"k": "a\ud800b"}: 500, with the episode recorded in semantic memory's three sets and in no other store;producer"a\u0000b": 500, nothing written;Why
MemoryMessagetypes these fields asstr(packages/common/src/memmachine_common/api/spec.py:458-477), andEpisodeEntrydoes too (packages/server/src/memmachine_server/common/episode_store/episode_model.py:29-35).MemMachine.add_episodeswrites the episode store, episodic memory, and semantic memory concurrently and raises the first failure once all of them return (packages/server/src/memmachine_server/main/memmachine.py:795-819), so the stores that keep the value have written it.Expected
The request refuses such a string with 422 before any store writes, and building an
EpisodeEntrywith one raises.Fix
#1792 types these fields as
PropertyStr, which refuses a string with U+0000 or a surrogate code point.🤖 Written by Claude Code (Claude Opus 5.5) on behalf of @edwinyyyu.