Skip to content

fix(gfql): explicit polars-gpu runs the Cypher OLAP fast paths on GPU or raises - #1979

Merged
lmeyerov merged 1 commit into
masterfrom
fix/1824-polars-gpu-fastpath-contract
Aug 19, 2026
Merged

lmeyerov merged 1 commit into
masterfrom
fix/1824-polars-gpu-fastpath-contract

Conversation

@lmeyerov

Copy link
Copy Markdown
Contributor

Fixes #1824

The rule being implemented

Owner ruling: auto may switch modes; an explicit engine selection may not. An explicitly requested engine either runs as requested or raises — it is never silently run somewhere else and reported as the requested engine.

What was broken (verified on current master, af494becc)

The issue's line references have moved. Re-read against master:

issue reference current master
Engine.py:33 POLARS_ENGINES = (Engine.POLARS, Engine.POLARS_GPU) graphistry/Engine.py:40
lazy/__init__.py:186-198 NO-CHEATING contract graphistry/compute/gfql/lazy/__init__.py:233-255 (_engine_for), error text at :258-266
gfql_fast_paths.py:1688/1868/2036 requested_engine = resolve_engine(...) graphistry/compute/gfql_fast_paths.py:2371 / 3039 / 3325
gfql_fast_paths.py:1070/1449/1519/1713/1907 if requested_engine in POLARS_ENGINES: graphistry/compute/gfql_fast_paths.py:1077 / 1463 / 1540 / 1621 / 1699
gfql_fast_paths.py:871,878 bare .collect() already routed through gfql.lazy.collect; the module has zero bare .collect() calls today (the one grep hit at :2112 is inside a comment)

So the collect-routing half of #1824 had landed — but gfql.lazy.collect picks its engine from active_target(), and the fast-path routes never set that target to GPU:

  • gfql_unified.py:1373 was a function named _fast_path_execution_target_ignoring_requested_engine that returned ExecutionTarget.CPU unconditionally ("Not GPU until every fast-path arm is GPU-or-decline");
  • _apply_connected_match_join's two-star arms (_connected_join_two_star_fast_grouped_count, _connected_join_two_star_fast_rows, and the fused lane behind them) were a second call site that was never wrapped in a target at all.

Result: every fast-path-served OLAP shape collected on CPU polars while the caller had asked for — and any benchmark would label — GPU.

AUTO can now reach POLARS_GPU — the issue's premise is partially refuted

Engine.py:34 still says "Explicit opt-in only — AUTO never selects it", and resolve_engine never returns POLARS_GPU for AUTO. But gfql_unified.py:2298/2323/2426 (the owner-directed cuDF arm, 2026-08-02) re-enters gfql() with engine=_AUTO_CUDF_ROUTE_ENGINE == 'polars-gpu' when every bound frame is cuDF and the GPU target probes usable. So a resolved POLARS_GPU no longer proves the user typed it.

Testing the resolved engine is still exactly right, and still exactly the owner's rule: that arm wraps its recursion in except NotImplementedError -> legacy CUDF path, so the mode switch happens in the AUTO layer (allowed) while the inner run stays GPU-or-raise (required). Values are unchanged either way — see the evidence below.

The fix

One shared seam rather than per-site edits, so the ~5 lazy collect sites cannot drift apart:

  • _fast_path_execution_target(engine) — GPU iff the requested engine is polars-gpu, CPU otherwise.
  • _run_fast_path_on_requested_target(engine, run) — runs a fast-path arm under that target and returns (result, reason). On the GPU target a non-GPU-executable plan surfaces as NotImplementedError (via lazy._gpu_raise) and becomes a decline, so the generic route — itself GPU-or-raise — answers. On the CPU target an NIE stays a real error.

Both call sites now go through it: _try_fast (same_path / row_pipeline arms) and _apply_connected_match_join's two two-star arms.

Evidence

Measured on a real GPU (RTX 3080 Ti, cudf 25.10, cudf-polars-cu12 25.10 + polars 1.32.3), on the fused two-star lane — the graph-benchmark q7 anatomy:

fused-lane collect target GPUEngine collects
master, engine='polars-gpu' cpu 0
this PR, engine='polars-gpu' gpu 1
master & this PR, engine='polars' / 'auto' cpu 0

Values identical in every cell.

Red at merge-base: the new pins were re-run at af494becc with one shim line (_fast_path_execution_target = _fast_path_execution_target_ignoring_requested_engine) and everything else byte-identical — 9 failed / 26 passed with the GPU stack, 8 failed / 26 passed / 1 skipped on a CPU-only box (the real-GPU pin skips). Green here: 35 passed (GPU), 34 passed + 1 skipped (CPU-only).

Per-site mutation (all non-vacuous): target decision forced back to CPU -> 9 red; _try_fast site unwrapped -> 2 red; connected-join grouped-count arm unwrapped -> 3 red; connected-join rows arm unwrapped -> 2 red.

Failure-set comparison: bin/test-polars.sh (serial) at merge-base and at this head, same box, same env — identical failure SETS in both directions (5 pre-existing, unrelated: 3 cuDF degree-consult, 1 viz categorical, 1 dask coercion). 7304 passed here vs 7286 at master (the delta is the new pins).

polars / auto unchanged: covered by the identical failure sets above; by the CPU-target and NIE-is-a-real-error pins parametrized over polars and auto; and by an explicit A/B of engine='auto' on a cuDF graph (the arm that routes through POLARS_GPU) against cudf and pandas — byte-identical results at master and at this head.

bin/ci_comment_density_guard.py, bin/lint.sh, bin/typecheck.sh all rc=0. Correctness only; no performance measurements were run. Expect polars-gpu to get slower on these shapes — the old numbers were CPU numbers wearing a GPU label.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AjbKuKheqDu78oapRT5AYm

… or raises

The fast-path call sites pinned the lazy execution target to CPU whatever engine
the caller asked for, and the connected-match-join two-star arms were never
wrapped in a target at all. Every fast-path-served OLAP shape therefore collected
on CPU polars under an `engine='polars-gpu'` label -- the exact mislabelling
`lazy._engine_for`'s `raise_on_fail=True` NO-CHEATING contract exists to prevent,
and undetectable downstream.

One shared seam, `_run_fast_path_on_requested_target`, now runs every fast-path
arm on the requested engine's target, so the five lazy collect sites in
`gfql_fast_paths` cannot drift apart. A plan node cudf-polars cannot execute
surfaces as `NotImplementedError` and is treated as a fast-path DECLINE; the
generic route -- itself GPU-or-raise -- answers, so nothing is quietly served on
CPU. `polars`, `auto`, `pandas` and `cudf` keep the CPU target, and an NIE there
is still a real error rather than a decline.

Fixes #1824

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AjbKuKheqDu78oapRT5AYm
@lmeyerov

Copy link
Copy Markdown
Contributor Author

Second failure-set comparison, added after the PR body was written — the full graphistry/tests/compute tree (not just the polars lane), serial, same box, same env, merge-base af494becc vs this head:

result
merge-base 7 failed, 12385 passed, 958 skipped, 46 xfailed
this head 7 failed, 12403 passed, 959 skipped, 46 xfailed

Failure SETS identical in both directions — zero new, zero accidentally fixed. All 7 are pre-existing and unrelated to this change:

gfql/cypher/test_lowering.py::test_t6_per_type_facts_engage_for_cypher_label_syntax[cudf]
gfql/cypher/test_lowering.py::test_t6_per_type_facts_prove_typed_bounds_whole_frame_cannot[per_type-True-expect_hint2-cudf]
gfql/index/test_degree_consult.py::test_degree_fact_is_built_and_actually_used[cudf]
gfql/index/test_degree_consult.py::test_slice_is_exact_across_domain_shapes[2-8-cudf]
gfql/index/test_degree_consult.py::test_slice_is_exact_across_domain_shapes[3-3-cudf]
gfql/test_viz_pipeline_conformance.py::test_categorical_searchany_decline_or_correct
test_engine_coercion.py::TestChainCoercion::test_chain_dask_edges

The pass-count delta (+18) is the new pins in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GFQL: Cypher OLAP fast paths silently run on CPU under engine='polars-gpu' (bare LazyFrame.collect() bypasses the GPU-or-error contract)

1 participant