What happened
Three backends keep state in process memory that other replicas change in the database.
Neo4jSemanticStorage loads _set_embedding_dimensions once at startup (semantic_memory/storage/neo4j_semantic_storage.py:1354-1364), and vector search iterates only the sets in that map (:1015-1030). A set whose first features were written by another replica is silently absent from this replica's results until this replica writes to the set or restarts. _drop_set_id_vector_index (:199-216) drops the index and the SetEmbedding node but clears only local caches, so other replicas keep querying a dropped index.NebulaGraphVectorGraphStore._graph_type_schemas (common/vector_graph_store/nebula_graph_vector_graph_store.py:158) is never loaded from the database. The first batch a process sees issues ADD NODE TYPE IF NOT EXISTS from that batch's properties and caches it as the truth (:1749-1762). If another process created the type with a different property set, the statement is a no-op, the cache lists properties the type lacks, and this process's INSERTs carry columns that do not exist. A single process after a restart hits the same path.Neo4jVectorGraphStore fills _index_state_cache from SHOW INDEXES only while it is empty (common/vector_graph_store/neo4j_vector_graph_store.py:934-940) and keeps approximate per-process node counts (:202-209) that decide when to create indexes. An index created elsewhere is invisible, so search stays exact (:496-503). Correctness is unaffected because the DDL is IF NOT EXISTS; the cost is performance.
Expected
These backends stay available for single-replica deployments and are refused, not silently wrong, in multi-replica ones. A declared concurrency scope per component does that without a routing layer. Routing by session does not keep these caches coherent: semantic sets can be org-level and shared across projects (#1732), so affinity by project does not pin a set to one replica, and a failover loses the cache anyway.
Notes
Code read at d6068cdbf (main), paths under packages/server/src/memmachine_server/. #1574 lists the declarative backend as out of scope for the storage overhaul; this issue records the facts so the scope decision is explicit.
🤖 Written by Claude Code (Claude Fable 5.1) on behalf of @edwinyyyu.
What happened
Three backends keep state in process memory that other replicas change in the database.
Neo4jSemanticStorageloads_set_embedding_dimensionsonce at startup (semantic_memory/storage/neo4j_semantic_storage.py:1354-1364), and vector search iterates only the sets in that map (:1015-1030). A set whose first features were written by another replica is silently absent from this replica's results until this replica writes to the set or restarts._drop_set_id_vector_index(:199-216) drops the index and theSetEmbeddingnode but clears only local caches, so other replicas keep querying a dropped index.NebulaGraphVectorGraphStore._graph_type_schemas(common/vector_graph_store/nebula_graph_vector_graph_store.py:158) is never loaded from the database. The first batch a process sees issuesADD NODE TYPE IF NOT EXISTSfrom that batch's properties and caches it as the truth (:1749-1762). If another process created the type with a different property set, the statement is a no-op, the cache lists properties the type lacks, and this process's INSERTs carry columns that do not exist. A single process after a restart hits the same path.Neo4jVectorGraphStorefills_index_state_cachefromSHOW INDEXESonly while it is empty (common/vector_graph_store/neo4j_vector_graph_store.py:934-940) and keeps approximate per-process node counts (:202-209) that decide when to create indexes. An index created elsewhere is invisible, so search stays exact (:496-503). Correctness is unaffected because the DDL isIF NOT EXISTS; the cost is performance.Expected
These backends stay available for single-replica deployments and are refused, not silently wrong, in multi-replica ones. A declared concurrency scope per component does that without a routing layer. Routing by session does not keep these caches coherent: semantic sets can be org-level and shared across projects (#1732), so affinity by project does not pin a set to one replica, and a failover loses the cache anyway.
Notes
Code read at
d6068cdbf(main), paths underpackages/server/src/memmachine_server/. #1574 lists the declarative backend as out of scope for the storage overhaul; this issue records the facts so the scope decision is explicit.🤖 Written by Claude Code (Claude Fable 5.1) on behalf of @edwinyyyu.