Repository navigation
[Bug]: Neo4j performance warnings: Cartesian product in Cypher queries during load testing #906
Description
Activity
Based on my initial analysis, it seems highly likely that the performance warnings (Cartesian product alerts and slowdowns) observed during bulk/batch ingestion in Neo4j are connected to the delayed creation of an index on the uid property. Here’s my hypothesis with known Neo4j and graph database behavior:
Key points:
- The codebase uses a threshold (default 10,000 nodes) before it creates an index on uid.
- Until that threshold is reached, Cypher queries matching nodes (e.g.,
MATCH (n {uid: ...})) must perform a full scan of all nodes with the label, i.e., an$O(N)$ operation. - When using bulk operations with many
MATCHclauses and disconnected patterns, this leads to full scans for each source/target lookup prior to the CREATE, easily causing cartesian products, excessive data generation, and slow query planning. - As the node count grows, each new write gets slower due to the lack of index support.
Effects:
- The
UNWIND+MATCH (source ...), (target ..)pattern produces a cartesian product warning if the two patterns are not connected by a variable in the MATCH. This is worsened if there are no indexes to efficiently look up nodes by uid. - Before hitting the node threshold, every write is highly inefficient; after that, index creation should rapidly improve per-write speed and reduce cartesian product risk.
Next steps for investigation:
- Locate the threshold and index creation logic: Find where in the codebase the logic checks node count and creates an index on
uidonce >10,000 nodes exist. - Review Cypher query construction: See if the pattern could be revised to connect the source/target lookups and avoid accidental cartesian products, e.g., by using a single MATCH with both nodes linked by an edge variable.
- Potential improvement: Consider lowering the threshold for index creation or checking if the deployment can support up-front index creation to reduce early insert cost and noisy warnings drastically.
Suggested next steps:
- Find the exact file and line where this threshold is enforced and how index creation is triggered
- Research suggestions for safer bulk Cypher query patterns
Describe the bug
During load testing, see #837, the docker logs from the memmachine-app container show repeated Neo4j performance notifications indicating potential inefficiency. The logs report that Cypher queries trigger cartesian product warnings due to disconnected patterns in the MATCH clause, which may cause excessive data generation and slowdowns during query processing.
Steps to reproduce
[INFO] neo4j.notificationslines with messages about cartesian products, example:Expected behavior
Cypher queries should avoid producing cartesian products unless intended. Queries should match only related node pairs, and logs should not contain unexpected performance warnings. Overall load test performance should not be negatively impacted by inefficient query planning.
Environment
Additional context
Docker logs from the memmachine-app container repeatedly show Neo4j cartesian product performance notifications like:
These appear after running bulk/batch ingestion and do not halt operation, but suggest query optimization opportunities.