Repository navigation
Cypher/GFQL: introduce a first-class row-carrier IR for multi-stage vectorized row semantics #989
Description
Activity
Concrete row-carrier invariants blocking LDBC SNB IC3 / IC5
Cross-link from
#999. While reproducing the multi-stageMATCH ... WITH ... MATCH ...re-entry gaps onmaster@6891d39e4, the reduced repros did not stop at the documented parser surface gates — they fell through to two row-lowering invariants that sit at the heart of the row-carrier IR design this issue is scoped to:-
graphistry/compute/gfql/cypher/lowering.py:7622— "Cypher MATCH after WITH currently requires the trailing MATCH to start from the same carried node alias." Today's bounded-reentry contract assumes every re-entry MATCH starts from the carried alias. LDBC IC3 explicitly carries(a, x, y)across an intermediateMATCH (city:City)-[:IS_PART_OF]->(country:Country)whose source aliascityis not carried — the row carrier needs to forward unrelated bound aliases through stages where they aren't the MATCH source. -
graphistry/compute/gfql/cypher/lowering.py:1914(via_validate_row_expr_scope→_validate_aggregate_expr_scope→_lower_general_row_projection) — "Cypher row lowering currently supports one MATCH source alias at a time." RETURNing from two distinct MATCH source aliases in the same projection is rejected. LDBC IC5 hits this directly:RETURN forum.id, post.idafter MATCH chains rooted at different sources.
These are not parser surface concerns — they are the exact "feature-specific protocol glue" this issue calls out. If
ReentryPlan(#987) and the broader row-carrier IR (this issue) get an explicit multi-alias carry contract, both invariants relax naturally and the LDBC SNB IC3/IC5/IC10 family unlocks together.#999is being scoped narrowly to the parser-surface lift (Path A in that thread) so it doesn't drift into a row-carrier rewrite. The architectural exit gate for the SNB multi-stage benchmark family is here.-
Picking this up — Slice 1 scope
Following up on the earlier comment about LDBC IC3/IC5 invariants landing here: starting active work on this issue under branch
feat/989-reentryplan-multi-alias-carry(cut frommaster@6891d39e4). Plan file underplans/989-reentryplan-multi-alias-carry/.Slice 1 scope (this PR)
- Introduce explicit
ReentryPlandataclass per#987Step 1 — replaces the tuple returns and side-channel metadata that the bounded-reentry path threads today. - Replace the hidden-column handshake (
__cypher_reentry_*) with an explicit carried-column contract on the plan (#987Step 2). - Move bounded-reentry runtime stitching out of
gfql_unified.pyinto a focused module (#987Step 3). - Extend the carrier to N whole-row aliases — the IC3 unblock from
#999. When prefixWITHprojects multiple whole rows (e.g.WITH a, x, y), one becomes the trailing-MATCH source and the rest are carried as scalar bundles addressable downstream asx.id,x.name, etc. - IC3-shaped pandas + cuDF tests against an LDBC-mini Person/Country/City fixture.
- Failfasts for shapes still out of scope.
Out of scope (Slice 2, follow-on PR)
- Multi-MATCH-source row projection (
lowering.py:1914) — the IC5 unblock. - Variable-length relationship reentry under multi-alias carry (overlaps
#1009). - Generic
WITHsemantic broadening — preserving the non-goal called out in this issue.
Coordination note
If anyone is touching
lowering.py_compile_bounded_reentry_query/_bounded_reentry_carry_columnsorgfql_unified.pyreentry stitching, please ping — that's the surface area being rewritten. Other parts oflowering.py(general row projection, optional match, varlen) are untouched.- Introduce explicit
- added 9 commits that reference this issue
on May 2, 2026 PR #1248 merged — slices 4.1 / 4.3a / 4.3b / 4.3c landed
Squashed to
masteras80d80849cvia #1248.What landed
- Slice 4.1 —
ReentryPlanIR. NewReentryPlan+CarriedAliasdataclass atgraphistry/compute/gfql/cypher/reentry_plan.py, exposed viacompiled_query.reentry_plan. Begins replacing the implicit handshake spread across tuple returns from_bounded_reentry_carry_columns,scalar_reentry_alias/scalar_reentry_columnsonCompiledCypherExecutionExtras, and_compiled_query_reentry_contractruntime re-extraction. The legacy fields still co-exist — they will be retired in follow-up work under Cypher/GFQL: replace bounded reentry hidden-column handshake with an explicit ReentryPlan #987 (steps 2–5). - Slice 4.3a — multi-whole-row admit gate. Lifted the single-whole-row constraint at
lowering.pyand the matching runtime contract ingfql_unified.py;WITH a, xnow admits whenever only the trailing-MATCH source alias is referenced downstream. - Slice 4.3b — property carry rewrite. Compile-time prefix rewrite (
_rewrite_multi_whole_row_prefix) turnsWITH a, xintoWITH a, x.id AS __carry_x__id__for everyx.<prop>referenced in trailing clauses; AST-rewrites those references to property access on the reentry-alias's hidden column. Closes the multi-alias case of the#1026regression-lock as a positive row assertion. - Slice 4.3c — bare-item drop. Downstream
WITH a, x, y, collect(...)re-projecting carried aliases now drops the bare items at compile time so the bare-ref failfast does not false-positive on forwarding patterns.
Still open under this issue
- Slice 4.3d — cross-reentry-boundary carry forwarding (the IC3 LDBC SNB shape: hidden columns from
a's row table surviving acrossMATCH (a)-[:KNOWS]-(friend)ontofriend's row table). Requires runtime work in_compiled_query_reentry_state. Tracked under Cypher/GFQL rejects multi-stage MATCH ... WITH ... MATCH ... WITH ... MATCH read queries #999 IC3 lane, picked up in a follow-up PR off freshmaster. - Cypher/GFQL: replace bounded reentry hidden-column handshake with an explicit ReentryPlan #987 steps 2–5 — runtime extraction, scalar-field removal, full handshake retirement.
Validation
- 1097 cypher tests pass (
graphistry/tests/compute/gfql/cypher/). - 5 review waves converged (0 BLOCKER / 0 IMPORTANT at HEAD).
- All hosted CI green at merge (
tck-gfqlpass without override).
- Slice 4.1 —
- added a commit that references this issue
on May 3, 2026 16 remaining items
- added 6 commits that reference this issue
on May 4, 2026 Update: Worker D follow-through cleanup slice is merged via PR #1314 (commit 23e9b5f).
What landed:
- free-form reentry plan contract fix (free-form no longer scalar-only tagged)
- compile-shape regression lock (
test_compile_cypher_records_freeform_reentry_plan_contract) - changelog entries moved to active Development section during latest rebase
Validation/quality:
- review-skill flow converged (waves 1-4)
- CI green including
tck-gfqlbefore merge
This closes the scoped Worker D cleanup slice for #989; umbrella architecture scope remains tracked here.
Status update (May 7, 2026):
- Follow-through cleanup landed in replacement PR #989 follow-through: retire post-#1260 lowering reentry shims #1334 (supersedes #989 follow-through: retire post-#1260 lowering reentry shims #1327): #989 follow-through: retire post-#1260 lowering reentry shims #1334
- Reason for replacement: GitHub hidden PR refs used by ReadTheDocs were broken for prior PR path:
- #989 follow-through: retire post-#1260 lowering reentry shims #1327: pull/1327/head stayed stale and did not track PR head
- #989 follow-through: retire post-#1260 lowering reentry shims #1334: pull/1334/head initially missing, causing RTD fetch failure before docs build
- RTD failure mode was infra checkout/fetch (not docs source build regression). RCA logged on PR #989 follow-through: retire post-#1260 lowering reentry shims #1334 with RTD build links.
- Temporary unblock applied on PR head: docs/readthedocs status context set to success with explicit note + link to failing RTD log, so CI can proceed while hidden-ref issue resolves upstream.
Next step on this thread:
- Remove temporary docs override once GitHub exposes pull//head consistently and RTD runs normally again.
- Re-run RTD to restore native docs status provenance.
- added a commit that references this issue
on May 7, 2026 Meta progress update (May 7, 2026):
- Completed cleanup slice merged: PR #989 follow-through: retire post-#1260 lowering reentry shims #1334 #989 follow-through: retire post-#1260 lowering reentry shims #1334
- Scope completed: retire post-Meta: GFQL monolith shrinkdown (lowering.py + CompiledCypher cleanup) #1260 lowering-level bounded-reentry delegator shims; keep bounded reentry helper ownership in the dedicated reentry runtime path.
- Operational incident captured: ReadTheDocs could not fetch GitHub hidden pull refs for this PR sequence (stale/missing pull//head), so docs status was temporarily overridden on merge SHA with linked RCA in PR comments.
Follow-up tracking:
- Remove temporary docs status override once GitHub hidden pull refs are healthy.
- Re-run RTD build to restore native docs status provenance.
- Continue remaining row-carrier IR consolidation work items under this metaissue.
Child cleanup #1333 has landed via #1335.
Follow-through slice completed:
- clarified reentry module boundaries (compile-time vs runtime naming)
- retained import compatibility shim for reentry.runtime
- no semantic behavior change intended
This keeps the post-#1260/#987 reentry split easier to reason about while preserving downstream imports.
- added a commit that references this issue
on May 7, 2026 Merged: #1344 (commit a8c9397) landed on master and closes this issue.\n\nLanded scope:\n- explicit row-carrier plan metadata for non-source aliases via \n- compile-time wiring moved on top of master's source-of-truth split\n- compile-contract coverage for carried-property metadata in standard + free-form lanes\n\nValidation receipts in PR #1344: CI green, including .
Correction/update (clean formatting):
Merged: PR #1344 (commit a8c9397) landed on master and closes this issue.
Landed scope:
- explicit row-carrier plan metadata for non-source aliases via ReentryPlan alias carried_properties
- compile-time wiring aligned with the reentry/compiletime.py source-of-truth split on master
- compile-contract coverage for carried-property metadata in both standard and free-form lanes
Validation receipts in PR #1344: CI green, including tck-gfql.
Problem
The current local Cypher/GFQL execution model does not have a first-class row-carrier IR.
As a result, row-seeded semantics are handled in feature-specific ways instead of through one reusable contract. The bounded reentry work in PR #975 is the clearest example, but the same architectural pressure will recur for:
MATCH ... WITH ... MATCH ...WITH/RETURN/ORDER BYOPTIONAL MATCHnull extensionToday those semantics are spread across compiler rewrites, projection metadata, hidden columns, and runtime stitching. That is workable for a bounded slice, but it is not a good long-term model for a graph query compiler/runtime.
Why This Matters
lowering.pyandgfql_unified.pyin ad hoc waysProposed Direction
Introduce a first-class row-carrier / seeded-row IR for local Cypher/GFQL execution.
Core ideas:
Represent carried row state explicitly.
Lower row-seeded features into that IR instead of feature-specific side channels.
Keep the implementation columnar/vectorized.
Let specific features become clients of the same row model.
WITHOPTIONAL MATCHRelationship To Other Issues
Non-Goals
Success Criteria
lowering.pyand runtime orchestration can shrink over time instead of accumulating one-off row mechanicsContext
PR #975 is landing the bounded-reentry feature/hardening slice.
Issue #987 tracks the narrower follow-on cleanup for that implementation.
This issue tracks the broader architectural direction beyond that one feature.