Repository navigation
GFQL OPTIONAL MATCH join materializes full cross-product before outer-scope filter (IS7 execution kill) #1052
Description
Activity
Rebaseline update: IS7 still OOM after #1061 (2026-04-05)
Context: pyg-bench branch
perf/rebaseline-2026-04-05, pincd707f0a(0.54.0).pygraphistry#1061fixed theOPTIONAL MATCH + CASEparser — IS7 no longer errors at parse time. But the process is still OOM-killed (returncode 137) on sf1 on dgx-spark CPU.What changed: Parse error → execution reach. What did not change: Memory footprint.
The OPTIONAL MATCH join still materializes a full cross-product before applying the outer-scope filter. On sf1 that's still ~120 GB RSS → SIGKILL. The parser fix is a necessary precondition but the semi-join reduction described in this issue is still the actual fix needed.
Priority: Highest on the CPU conformance backlog — IS7 is the only query that fully kills the process. IC6, IC9, etc. run (with gaps); IS7 does not run at all.
Full analysis:
docs/perf-analysis-rebaseline-2026-04-05.mdin pyg-bench branchperf/rebaseline-2026-04-05.- added a commit that references this issue
on Apr 6, 2026 Fix in PR #1067. Semi-join filter applied in
_apply_connected_optional_match(gfql_unified.py) restricts the optional arm to join-key values present in the base result before the left-outer-join. Eliminates the intermediate cross-product that caused the SIGKILL on sf1.- added a commit that references this issue
on Aug 15, 2026
Summary
The official benchmark lane for a short query involving
MATCH + OPTIONAL MATCH + CASE WHENkills the process with SIGKILL (returncode 137) on every tested pin and branch through current master. The query now lowers correctly — the remaining problem is in execution/resource territory during the probe itself.The suspected root cause is that the OPTIONAL MATCH join materializes a large intermediate result before applying the outer-scope binding filter. This is reproducible across multiple upstream pins on the staged sf1 dataset.
Status
Tested across five different pins/branches — all killed during the direct Cypher probe. Fixture load always completes cleanly. The kill happens inside query execution, not setup.
Full artifact table, RSS measurements, and fix direction are in the private benchmark tracker:
graphistry/pyg-bench#1.Fix direction
The OPTIONAL MATCH join should be scoped to the outer-scope binding variable before materialization rather than joining broadly and filtering after. Details in
graphistry/pyg-bench#1.