Skip to content

fix(gfql): stop WITH re-entry mutating the caller's Plottable (#1786) - #1793

Merged
lmeyerov merged 1 commit into
masterfrom
fix/gfql-1786-reentry-caller-mutation
Jul 27, 2026
Merged

lmeyerov merged 1 commit into
masterfrom
fix/gfql-1786-reentry-caller-mutation

Conversation

@lmeyerov

Copy link
Copy Markdown
Contributor

Fixes #1786.

The defect

g = graphistry.nodes(N, "id").edges(E, "s", "d")
g.gfql("MATCH (a {kind:'a'}) WITH a MATCH (a)-[*]->(b) RETURN count(*) AS c")  # 7 (correct)
g.gfql("MATCH (a)-[*]->(b) RETURN count(*) AS c")                              # 7  <-- was WRONG, is 13

WITH re-entry wrote its seed onto the caller's Plottable and never cleared it, so the next -- entirely unrelated -- query on the same object was answered against the stale seed. Fails open: no error, just the previous query's count. Engine-independent, so an engine A/B never surfaces it.

Leak sites (all _gfql_* fields assigned during execution were audited)

site what leaked caller-visible today?
gfql_unified._execute_compiled_query_chain_non_union _gfql_start_nodes yes -- _seeded_dispatch_graph hands back base_graph ITSELF when the compiled query has no seed rows, so the write lands on the user's graph. This is the reported bug.
gfql_unified.gfql (dispatch_self = self) _gfql_shortest_path_backend yes -- a per-CALL argument persisted on the caller's graph and became the default for its next query.
chain._handle_boundary_calls _gfql_start_nodes, _gfql_rows_base_graph no (its g_temp is always rebuilt by the middle ops first) -- same pattern, converted for uniformity.
polars.chain._run_calls_polars _gfql_start_nodes, _gfql_rows_base_graph not onto g, but onto the result on the all-calls path -- a follow-up query on that result is then answered against another graph's seed.

Left alone deliberately: row/pipeline.py's _gfql_native_sp_cache (an edges-identity-keyed memo, semantically neutral) and the compiled-string-query memo cache (now explicitly owned by self, so it still works).

Approach and why

Carry the state on an internal copy, not a try/finally restore. This mechanism already exists in the codebase -- gfql/index/handoff.py attaches its boundary decision with g.bind() -- so gfql/exec_context.py is the same shape for the row context, and every site now attaches on the way in and clears on the way out.

  • Threading the seed through ExecutionContext would be the purest fix, but it is read deep inside the pandas row pipeline and both polars pattern appliers; that is a large refactor, not a bug fix.
  • A finally restore was rejected: it still mutates a shared object for the duration of the call, and it cannot fix the second half of the defect -- the state escaping on the result graph.

Clearing on the way out matters independently: the result of a WITH query used to carry the seed, so a follow-up query on that result got the same wrong answer one hop removed.

No banned constructs (Any, cast, getattr, setattr, bare list, Dict[str, Any], object) -- the fields are already declared on Plottable, so it is ordinary typed attribute access throughout.

Tests

graphistry/tests/compute/gfql/test_reentry_caller_graph_immutability.py, 9 cases x {pandas, polars}: two independent queries on one graph, the re-entry answer itself, repeated re-entry, nested (two WITHs), order-independence, the caller's fields after a query, the result's fields, the pure-call chain result, and the shortest-path-backend argument.

Mutation-checked (each leak reintroduced in-container, suite re-run, then restored):

reintroduced new tests failing
dispatch_graph._gfql_start_nodes = start_nodes 8
dispatch_self._gfql_shortest_path_backend = ... 2
all three clear_row_exec_context calls 3
polars clear_row_exec_context alone 1
everything 13

The two attach sites that are not independently observable (chain / polars) are documented as such above rather than claimed as tested.

Verification

All on dgx-spark in graphistry/test-rapids-official:26.02-gfql-polars with --gpus all.

  • graphistry/tests/compute: master c76f23a3 = 9 failed, 7050 passed, 90 skipped, 16 xfailed; this branch = 9 failed, 7068 passed, 90 skipped, 16 xfailed. Identical failure set (7 igraph-cudf round-trip + 1 cudf coercion + 1 dask, all pre-existing); +18 = the new tests.
  • mypy: 161 errors on master, 161 on this branch -- zero added.
  • ruff: clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB

`g.gfql("MATCH (a {kind:'a'}) WITH a MATCH (a)-[*]->(b) RETURN count(*)")`
left the re-entry seed on `g` itself, so the NEXT, entirely unrelated query on
the same object was answered against it -- no error, just the previous query's
count. Engine-independent (wrong on pandas too, so an engine A/B never surfaces
it) and a direct violation of the repo's pure-functional contract: users
reasonably reuse one Plottable for many queries.

Leak sites (every `_gfql_*` field assigned during execution was audited):

  * gfql_unified `_execute_compiled_query_chain_non_union` -- the reported bug.
    `_seeded_dispatch_graph` returns `base_graph` ITSELF when the compiled query
    has no seed rows, so `dispatch_graph._gfql_start_nodes = ...` wrote straight
    onto the user's graph.
  * gfql_unified `gfql()` -- `dispatch_self = self` then
    `_gfql_shortest_path_backend = shortest_path_backend`: a per-CALL argument
    persisted on the caller's graph and silently became the default for its next
    query.
  * chain `_handle_boundary_calls` and its polars twin `_run_calls_polars` --
    same in-place writes of `_gfql_start_nodes` / `_gfql_rows_base_graph`. Their
    target is an internal graph on today's paths rather than the caller's, but
    they are the same pattern one refactor away from the same bug, and without
    the clear below the polars one rides out on the RESULT.

Approach: carry the state on an INTERNAL COPY, not a try/finally restore. The
per-execution context idea already exists here -- `gfql/index/handoff.py`
attaches its boundary decision with `g.bind()` -- so `gfql/exec_context.py` is
the same shape for the row context, and every site now attaches on the way in
and clears on the way out. Threading the seed through `ExecutionContext` instead
would be the purest fix but it is read deep inside the row pipeline and both
polars pattern appliers, so it is a large refactor; a `finally` restore was
rejected because it still mutates a shared object for the duration of the call
(anything holding `g` concurrently sees the seed) and it cannot fix the second
half of the defect, where the state escapes on the RESULT graph and poisons
queries run against that.

Clearing on the way out matters independently: the result of a WITH query used
to carry the seed, so a follow-up query on that result -- a different graph
entirely -- got the same wrong answer one hop removed.

Mutation-checked: reverting the gfql_unified seed write fails 8 of the new
tests, reverting the shortest-path-backend write fails 2, removing the three
clears fails 3, and a full revert fails 13. Tests run on both pandas and polars
since the defect is engine-independent.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB
@lmeyerov
lmeyerov merged commit 233b64c into master Jul 27, 2026
77 checks passed
lmeyerov added a commit that referenced this pull request Jul 28, 2026
…failure

The first version of this file's engine probe wrapped a smoke query in a bare
`except Exception: return False`. Two things went wrong with that, both found by
trying rather than by argument (on the sibling #1788/#1790 parametrization):

  * a probe that runs THE SHAPE UNDER TEST disarms its own file — reverting a
    production guard made the probe raise and every parameter reported SKIPPED
    instead of failing;
  * a probe that swallows EVERYTHING is worse — a transient
    `MemoryError: ... cudaErrorMemoryAllocation` in a fresh GPU container
    silently dropped cuDF from a run that otherwise passed, and a skipped GPU
    parameter reads as evidence of passing.

So the check now CLASSIFIES: a missing module skips, a recognisable GPU-stack
error skips WITH ITS TEXT QUOTED in the reason, and any other failure
propagates. The smoke query stays a plain traversal, never the shape under test.

GPU receipts — dgx GB10, `graphistry/test-rapids-official:26.02-gfql-polars`,
`docker run --gpus all`, cuDF 26.02.01 / polars 1.35.2: 43 passed, 0 skipped
(this file plus the #1793 suite), repeated.

The marker list is duplicated from the copy landing in `polars_test_utils.py`
alongside the #1788/#1790 parametrization; collapse to one definition once both
land. Kept duplicated for now so the two PRs stay independently mergeable.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB
lmeyerov added a commit that referenced this pull request Jul 28, 2026
… review)

Review asked whether `clear_row_exec_context` should RESTORE the value the graph
carried on entry rather than NULLing it — a fair question, because
`attach_row_exec_context` INHERITS on the way in (a `None` argument keeps what
`g` already carries), so an outer scope's value can reach an inner execution and
be dropped by it.

Determination: NULL is correct and restore would reintroduce #1786. Three
reasons, each now carried by a test rather than by argument:

  1. `clear` is PURE. It returns `g.bind()` and never writes through to the
     object it was handed, so an outer scope that set the field still has it
     afterwards. There is no caller state to save. That is exactly what
     separates this from #1786, which WAS an in-place write onto the caller's
     own graph — restore is the fix for a mutation, and there is no mutation.
  2. The only channel restore would change is the RETURN VALUE, and putting the
     seed back there IS the second half of #1786 ("the result of a WITH query
     carries the seed, and a follow-up query on that result is answered against
     it"). Measured: hand-restoring the seed onto a result changes the answer of
     the next query on it (7 -> 2 -> 1 rows).
  3. No execution frame inherits a context it did not set. Instrumenting
     `attach` over `graphistry/tests/compute` recorded 3907 calls and ZERO
     inheriting ones. 53 DID enter on a graph already carrying a seed (nested
     boundary frames) but each was handed the IDENTICAL `start_nodes` parameter,
     so the frame still owns what it sets. The cross-segment WITH seed travels
     as the explicit `start_nodes` PARAMETER (`chain_impl(..., start_nodes=)`,
     `_compiled_query_reentry_state`), never through the graph field, so the
     field's lifetime is exactly one boundary-call run.

`test_exec_context_scoping.py` re-runs that ownership measurement as an
assertion over a corpus of the shapes that reach every attach site, so a future
path that starts relying on inheritance reopens this decision loudly instead of
silently losing an outer value.

MUTATION-CHECKED IN THE DECIDING DIRECTION: implementing save/restore at all
three attach sites (chain, native polars chain, gfql_unified) fails 4 of the new
tests on every runnable engine — while the existing #1793 suite
(`test_reentry_caller_graph_immutability.py`) passes UNCHANGED. Those tests
could not distinguish the two designs; this file can, which is why it exists.

Engine-parametrized over pandas/polars/cuDF/polars-gpu with a fixed engine list
plus a runtime probe (not `available_nonpandas_engines()`, which silently
shrinks): a non-runnable engine reports SKIPPED, it does not vanish from the
report. Also added to `bin/test-polars.sh` — the file has no module-level
`importorskip`, so nothing would otherwise have flagged that its polars params
run in no lane (#1795 class).

RUNTIME DELTA: zero. No production line changed; the only non-test edits are the
`clear_row_exec_context` docstring and a CHANGELOG entry, so no pyg-bench lane
run is required (CB5).

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB
lmeyerov added a commit that referenced this pull request Jul 28, 2026
…failure

The first version of this file's engine probe wrapped a smoke query in a bare
`except Exception: return False`. Two things went wrong with that, both found by
trying rather than by argument (on the sibling #1788/#1790 parametrization):

  * a probe that runs THE SHAPE UNDER TEST disarms its own file — reverting a
    production guard made the probe raise and every parameter reported SKIPPED
    instead of failing;
  * a probe that swallows EVERYTHING is worse — a transient
    `MemoryError: ... cudaErrorMemoryAllocation` in a fresh GPU container
    silently dropped cuDF from a run that otherwise passed, and a skipped GPU
    parameter reads as evidence of passing.

So the check now CLASSIFIES: a missing module skips, a recognisable GPU-stack
error skips WITH ITS TEXT QUOTED in the reason, and any other failure
propagates. The smoke query stays a plain traversal, never the shape under test.

GPU receipts — dgx GB10, `graphistry/test-rapids-official:26.02-gfql-polars`,
`docker run --gpus all`, cuDF 26.02.01 / polars 1.35.2: 43 passed, 0 skipped
(this file plus the #1793 suite), repeated.

The marker list is duplicated from the copy landing in `polars_test_utils.py`
alongside the #1788/#1790 parametrization; collapse to one definition once both
land. Kept duplicated for now so the two PRs stay independently mergeable.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB
lmeyerov added a commit that referenced this pull request Jul 28, 2026
Both landed without a CHANGELOG entry. #1793 is the one that matters: it fixes a
SILENT WRONG ANSWER that is user-visible on every engine.

Reproduced on the pre-fix tree while auditing the merge, so the entry states a
measured effect rather than a description: `MATCH (a {grp:1}) WITH a MATCH
(a)-[]->(b) RETURN count(*)` followed by a plain `MATCH (a)-[]->(b) RETURN
count(*)` on the SAME graph object returned 60 instead of 134, 69 instead of 136
and 61 instead of 143 across seeds. On polars the poisoned graph then raised
`NotImplementedError` for EVERY subsequent query — a failed query left the
user's object broken. Both are fixed on master; neither was written down.

#1792's graphviz half (`KeyError: None` -> an actionable `ValueError` on a
bound-but-unlabelled frame) is smaller but is still a user-facing error-message
change, so it gets an entry too.

DOCS ONLY: no code touched, so runtime delta is zero and no pyg-bench lane run
is required (CB5) — checkable from the diff, which is one file.

NOT INCLUDED, deliberately. The audit also flagged #1792's `dtype: object` ->
`dtype: DType` as a typing WIDENING (`DType = Any`, which is strictly weaker
than `object`, and mypy is indifferent between them here — verified). I am not
reverting it: `graphistry/compute/typing.py` declares `DType` as the repo's
engine-agnostic dtype alias with the comment "Honestly Any -- the concrete type
is engine-dependent", and these parameters do receive pandas/cuDF/polars dtypes,
so the named alias is the documented convention even though it checks as `Any`.
Reverting on the audit's reading would churn against a stated convention on a
judgement call that belongs to the owner. Reported, not silently decided.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB
lmeyerov added a commit that referenced this pull request Jul 28, 2026
* test(gfql): decide #1793's clear-vs-restore question, and pin it (#1793 review)

Review asked whether `clear_row_exec_context` should RESTORE the value the graph
carried on entry rather than NULLing it — a fair question, because
`attach_row_exec_context` INHERITS on the way in (a `None` argument keeps what
`g` already carries), so an outer scope's value can reach an inner execution and
be dropped by it.

Determination: NULL is correct and restore would reintroduce #1786. Three
reasons, each now carried by a test rather than by argument:

  1. `clear` is PURE. It returns `g.bind()` and never writes through to the
     object it was handed, so an outer scope that set the field still has it
     afterwards. There is no caller state to save. That is exactly what
     separates this from #1786, which WAS an in-place write onto the caller's
     own graph — restore is the fix for a mutation, and there is no mutation.
  2. The only channel restore would change is the RETURN VALUE, and putting the
     seed back there IS the second half of #1786 ("the result of a WITH query
     carries the seed, and a follow-up query on that result is answered against
     it"). Measured: hand-restoring the seed onto a result changes the answer of
     the next query on it (7 -> 2 -> 1 rows).
  3. No execution frame inherits a context it did not set. Instrumenting
     `attach` over `graphistry/tests/compute` recorded 3907 calls and ZERO
     inheriting ones. 53 DID enter on a graph already carrying a seed (nested
     boundary frames) but each was handed the IDENTICAL `start_nodes` parameter,
     so the frame still owns what it sets. The cross-segment WITH seed travels
     as the explicit `start_nodes` PARAMETER (`chain_impl(..., start_nodes=)`,
     `_compiled_query_reentry_state`), never through the graph field, so the
     field's lifetime is exactly one boundary-call run.

`test_exec_context_scoping.py` re-runs that ownership measurement as an
assertion over a corpus of the shapes that reach every attach site, so a future
path that starts relying on inheritance reopens this decision loudly instead of
silently losing an outer value.

MUTATION-CHECKED IN THE DECIDING DIRECTION: implementing save/restore at all
three attach sites (chain, native polars chain, gfql_unified) fails 4 of the new
tests on every runnable engine — while the existing #1793 suite
(`test_reentry_caller_graph_immutability.py`) passes UNCHANGED. Those tests
could not distinguish the two designs; this file can, which is why it exists.

Engine-parametrized over pandas/polars/cuDF/polars-gpu with a fixed engine list
plus a runtime probe (not `available_nonpandas_engines()`, which silently
shrinks): a non-runnable engine reports SKIPPED, it does not vanish from the
report. Also added to `bin/test-polars.sh` — the file has no module-level
`importorskip`, so nothing would otherwise have flagged that its polars params
run in no lane (#1795 class).

RUNTIME DELTA: zero. No production line changed; the only non-test edits are the
`clear_row_exec_context` docstring and a CHANGELOG entry, so no pyg-bench lane
run is required (CB5).

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB

* test(gfql): classify engine availability instead of swallowing every failure

The first version of this file's engine probe wrapped a smoke query in a bare
`except Exception: return False`. Two things went wrong with that, both found by
trying rather than by argument (on the sibling #1788/#1790 parametrization):

  * a probe that runs THE SHAPE UNDER TEST disarms its own file — reverting a
    production guard made the probe raise and every parameter reported SKIPPED
    instead of failing;
  * a probe that swallows EVERYTHING is worse — a transient
    `MemoryError: ... cudaErrorMemoryAllocation` in a fresh GPU container
    silently dropped cuDF from a run that otherwise passed, and a skipped GPU
    parameter reads as evidence of passing.

So the check now CLASSIFIES: a missing module skips, a recognisable GPU-stack
error skips WITH ITS TEXT QUOTED in the reason, and any other failure
propagates. The smoke query stays a plain traversal, never the shape under test.

GPU receipts — dgx GB10, `graphistry/test-rapids-official:26.02-gfql-polars`,
`docker run --gpus all`, cuDF 26.02.01 / polars 1.35.2: 43 passed, 0 skipped
(this file plus the #1793 suite), repeated.

The marker list is duplicated from the copy landing in `polars_test_utils.py`
alongside the #1788/#1790 parametrization; collapse to one definition once both
land. Kept duplicated for now so the two PRs stay independently mergeable.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB

---------

Co-authored-by: Claude Opus 5 <[email protected]>
lmeyerov added a commit that referenced this pull request Aug 2, 2026
…iation

docs(changelog): the two entries missing from #1792 and #1793
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GFQL: WITH re-entry mutates the caller's Plottable, silently corrupting the next query

1 participant