Skip to content

Property strings with U+0000 or a lone surrogate pass the add-memories request and are refused by some stores after others have written them #1793

Description

@edwinyyyu

What happens

Measured on main at ad8ff24, one value at a time through each store's own code, on SQLite and PostgreSQL where a store supports both, with Qdrant 1.17.0 (REST and gRPC) and Milvus 2.6.24 and Milvus Lite. The value is a string that becomes a property value: a metadata key or value, producer, produced_for, or role of a message, which are producer_id, producer_role, produced_for_id, and metadata of an EpisodeEntry.

store "a\u0000b" "a\ud800b" (lone surrogate)
episode store, SQLite stored metadata value stored; producer_id refused (UnicodeEncodeError)
episode store, PostgreSQL refused (UntranslatableCharacterError for the metadata JSONB, CharacterNotInRepertoireError for producer_id) refused
segment store, SQLite stored stored
segment store, PostgreSQL refused (UntranslatableCharacterError) refused (InvalidTextRepresentationError)
semantic memory set ids, SQLite stored refused (UnicodeEncodeError)
semantic memory set ids, PostgreSQL (pgvector and vector-store storage) refused (CharacterNotInRepertoireError) refused (DataError)
SQLiteVectorStore, SQLiteVecVectorStore stored stored; an equality filter on it raises UnicodeEncodeError
QdrantVectorStore, REST and gRPC stored refused (PydanticSerializationError, UnicodeEncodeError)
MilvusVectorStore, 2.6.24 and Lite stored; on Lite an equality filter on it raises MilvusException refused (DataNotMatchException)

Semantic memory receives the value inside a set id, for producer and for a metadata value whose key a set type names.

The request accepts both: JSON "\u0000" decodes to U+0000, and the route's JSON decoding turns "\ud800" into a lone surrogate. Through POST /api/v2/memories with the real stores (episode store, segment store, and semantic storage on PostgreSQL, and SQLiteVectorStore):

  • metadata {"k": "a\u0000b"} or {"k": "a\ud800b"}: 500, with the episode recorded in semantic memory's three sets and in no other store;
  • producer "a\u0000b": 500, nothing written;
  • with the same stores on SQLite, all three: 200, stored everywhere.

Why

  • MemoryMessage types these fields as str (packages/common/src/memmachine_common/api/spec.py:458-477), and EpisodeEntry does too (packages/server/src/memmachine_server/common/episode_store/episode_model.py:29-35).
  • MemMachine.add_episodes writes the episode store, episodic memory, and semantic memory concurrently and raises the first failure once all of them return (packages/server/src/memmachine_server/main/memmachine.py:795-819), so the stores that keep the value have written it.
  • PostgreSQL text and JSONB refuse U+0000, and UTF-8 encoding refuses a lone surrogate.

Expected

The request refuses such a string with 422 before any store writes, and building an EpisodeEntry with one raises.

Fix

#1792 types these fields as PropertyStr, which refuses a string with U+0000 or a surrogate code point.

🤖 Written by Claude Code (Claude Opus 5.5) on behalf of @edwinyyyu.

Activity

  1. added theissue type on Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions