Repository navigation
fix(polars): endpoint is_in shape errors on Polars 1.21 traversal #2082
Description
Polars1.21.0 raises ComputeError: shapes don't match ... in is_in comparison during endpoint-resolvability filtering. This affects seeded typed-hop and OPTIONAL traversal queries on the Polars CPU engine installed in the RAPIDS25 test image.
Confirmed on both unmodified master 77bcbd337ad87c3d139e1f6f61fd041fd1563720 and combined PR2073 follow-ups 80271526f21e80e395b96c4fdbcaac6957b79c8c: the exact same33 test cases fail with the same error class. This is baseline compatibility debt, not a regression introduced by #2077/#2078/#2079 or #2080.
Reproduction environment: DGX Spark, graphistry/test-rapids-official:25.02-gfql, Polars1.21.0, cuDF25.2.2, pandas2.2.3, Python3.12. Commands were run sequentially under shared GPU locks and verified source manifests.
TEST_CUDF=1 TEST_POLARS_GPU=0 python -m pytest -q \
graphistry/tests/compute/gfql/test_seeded_node_lookup_fastpath.py \
graphistry/tests/compute/gfql/test_optional_match_semantics.py \
-k polars --maxfail=0 --tb=shortRepresentative failing test: test_optional_varlen_arm_bound_endpoints_projects_bound_alias[polars]. Additional cases include indexed/scan seeded typed-hop property projections and categorical node columns. The master run reports33 failed,255 passed,261 deselected. Expanded candidate suite reports33 failed,929 passed,195 skipped. The same expanded candidate suite passes962 tests in RAPIDS26 and1045 in the Polars-GPU image; there are no cuDF test failures in the25 lane.
Root site: graphistry/compute/gfql/lazy/engine/polars/hop_eager.py, _keep_edges_with_both_endpoints_resolvable: universe = resolvable_ids.implode() followed by endpoint is_in(universe). Polars1.21 interprets this RHS incompatibly and reports expected lengths0/8/113/226 versus1. Blame traces the imploded-universe construction to 86013f47f0 (2026-08-15), with later endpoint casting in d0e23d881a (2026-08-19). These dates establish provenance of the current construction; they do not establish the first release affected.
Priority: P2 compatibility defect. Current Polars/RAPIDS26 paths work; older Polars traversal correctness coverage is affected. Do not silently skip these cases or substitute CPU execution for explicit GPU requests. A fix should exercise empty and nonempty universes, null endpoints, both directions, indexed/scan paths, categorical IDs and mixed matching/nonmatching endpoints on both supported Polars versions.
Related: #2074 (discovery and baseline separation), #1890 (GPU CI coverage), #1938 (existing is_in warning tracking), #2026 (endpoint filtering performance). This issue tracks the concrete old-version execution failure, rather than those broader coverage/performance tasks.
Evidence retained locally under plans/gfql-2073-followups/logs/: device-master2080/, device-baseline2080-v2/, master2080-baseline-failure-parity.json; immutable image metadata, full source/probe SHA256 manifests, JUnit case identities and complete stacks accompany the runs. Deferred from the existing PR2073 fix set because exact baseline comparison proves the independent pre-existing defect.
Current verification checkpoint
Refreshed exact master92ad8c0dbe0af7acc56923f42153594cf5ab30fc control reproduces the same33failure IDs/error types as the follow-up candidate. Final combinedffa52d8a expanded matrix:cuDF25image1053passed/215skipped/33baseline failures;cuDF26image1086passed/215skipped;PolarsGPU26image1301passed/0skipped. This supplements the historical77/802 evidence above; it does not establish first affected release. P2deferral remains unchanged; current parity artifact is logs/master2081-final/gate-and-baseline-audit.json under the local2074plan.
Activity
lmeyerov commentedon Sep 15, 2026
Fix ready for review: #2085 (
fix/gfql-2082-polars121-isin, base master 65c359b)Root cause, pinned. polars changed the
is_inright-hand-side contract in 1.28.0: below it a List-typed RHS (ids.implode()) is matched row-wise and must have the column's length, which is the reportedshapes don't match: expected N elements in 'is_in' comparison, got 1; from 1.28 the bare-Series RHS is deprecated instead (pola-rs/polars#22149). Pinned by a per-version venv sweep over 14 releases 1.21.0 … 1.35.2: the imploded RHS fails through 1.27.1 and passes from 1.28.0; the bare RHS passes on every release and warns from 1.28.0.Fix. One helper (
lazy/engine/polars/membership.py:id_set,is_in_ids) spells the RHS for the installed polars; all 14 Series-membership sites route through it —hop_eager(the site reported here),chain_specializations/{hotpaths,point_rows},pattern_apply, and twogfql_unifiedsites that still used the bare form and therefore still warned on >= 1.28 (#1938 item 5). On polars >= 1.28 the emitted expressions are unchanged, including the hoisted hop universe.The blast radius was wider than the 33 cases here. On exact master in
test-rapids-official:25.02-gfql(polars 1.21.0),graphistry/tests/compute/gfql -k polarsfails 2361 cases and 1792 of them are this idiom — test_engine_polars_chain 729, hop boundary matrix 180, hop-labels differential 173, hotpaths 144, point_rows 97, and so on. The 33 named here are the two files this issue happened to sample. With the fix: 0 of that idiom, 1871 master failures pass, 0 new failures.Verification (DGX Spark; exact trees shipped by
git archivewith sha256 manifests, shared benchmark lock,--memorycapped, sequential):25.02-gfql(1.21.0)-k polars: 2361 failedpolars-gpuparams cudf-polars 25.02 cannot execute, 0 polars-CPU26.02-gfql(1.35.2)26.02-gfql-polars,TEST_POLARS_GPU=1CI on the exact PR head (
62e5b95aa): run 34938849389 SUCCESS — 85/85 jobs green, including all 7test-polarscells (py3.9–3.14) and all 10gfql-routes-offreplays, which run the new tests again with each fast-path route disabled. Two jobs (test-dgl-cpu,test-full-ai (3.9)) were initially cancelled by their own 15-minute timeout inside dependency installation, before running any test; both passed on re-run, and neither touches anything this branch changes.Also on the exact head: ruff, the type-hygiene and comment-density guards all clean; whole-tree mypy identical to master (258 pre-existing errors on both, 0 new); and the polars coverage lane plus
coverage_audit --profile gfql-polarsreplicated, with the new module at 100% against its floor.Controls. A mutant that always implodes re-breaks ~100 tests on 1.21. A mutant that always uses the bare form is caught only by recording warnings, because polars emits that deprecation from Rust and prints it under
-W error— worth knowing for anyone writing similar pins. Beyond values, a static AST lock now fails on anyis_in(...implode())written outside the helper, since that spelling is green on every CI polars and fatal here; three mutations ofhop_eagerback to the pre-fix spelling each trip it.One relabel, stated explicitly. Two of the 33 (
test_categorical_node_columns_keep_parity_on_every_engine, "props from three aliases", scan+indexed) pass theis_infix and then fail on polars 1.21 only because the fast path's categorical column comes back with categories['f62']while the full path keeps all 300: polars 1.21 keeps the full rev-map through filter/join and trims it after filter→join, 1.35 trims everywhere. The test's_canonnow drops unused categories before comparing; values and categorical dtype still compare exactly. The category set is not a row contract.Out of scope, pre-existing, still failing on polars 1.21: 365
polars-gpuNotImplementedError plus 54 GFQLUnsupportedError (cudf-polars 25.02 cannot execute those plans; the 25.02 GPU lane is not claimed), 3Expr.round(mode=)(a polars >= 1.29 feature; thepolarsextra floor is 1.29), 4 row-op NotImplementedError, 4 harness artifacts.Limitations. First affected release is still unknown: the imploded spelling landed 2026-08-15 (86013f4), so any release containing it fails on polars < 1.28. polars 1.21 remains below the
polarsextra floor — this makes the CPU engine run there, it does not claim feature parity. Separately, replicating the coverage lane showedlazy/engine/polars/dtypes.pyat 87.30% against its 92.0 floor on unmodified master too, in the same image; this branch does not touch that file, so it is environmental (py3.13 with cuDF here vs py3.12 without in CI) and no baseline was changed for it.test(gfql/polars): #2082 prove the version gate is load-bearing, not …
Merge pull request #2085 from graphistry/fix/gfql-2082-polars121-isin
lmeyerov commentedon Sep 16, 2026
Merged as c9d7d53 via #2085.
Post-merge close-out:
c1959825b: run 35039348961 SUCCESS, 85/85 jobs, including all 7test-polarscells (py3.9–3.14, polars 1.43.0) and all 10gfql-routes-offreplays.test_the_version_gate_is_load_bearing_on_the_installed_polarsnow forces the imploded arm and asserts the consequence version-agnostically: below 1.28 it must raise this issue's ComputeError, from 1.28 it must agree value-for-value with the routed arm. Neither lane skips.pattern_applyx2, the reachablegfql_unifiedOPTIONAL prune) genuinely change from a bare Series RHS to an imploded one. Interleaved A/B on DGX (polars 1.35.2, 300k nodes / 2M edges, 30 samples per arm per workload): two-hop endpoint gate (the perf(gfql): undirected k-hop chain re-runs the seed-rediscovery rule and the endpoint-resolvability filter on every pass #2026 path) median 80.6 -> 78.1 ms, seeded chain 9.5 -> 9.2 ms, EXISTS 312.6 -> 316.3 ms. All distributions overlap, nothing trips a 5%-and-2ms flag. The EXISTS minimum at +2.6% was isolated with a micro A/B of exactly the changed expression over 2M rows: imploded is 0.7-1.8% faster than bare, so that delta was box contention, not the diff. perf(gfql): undirected k-hop chain re-runs the seed-rediscovery rule and the endpoint-resolvability filter on every pass #2026 preserved.Limitations unchanged: first affected release still unknown (the imploded spelling landed 2026-08-15 in 86013f4, so any release containing it breaks on polars < 1.28), and polars 1.21 remains below the
polarsextra floor of 1.29 — the CPU engine runs there, feature parity is not claimed.lmeyerov commentedon Sep 16, 2026
Closing the one limitation this issue was left with — the first affected release is 0.59.0, and it is the only affected release.
86013f47f(2026-08-15) introduced the imploded RHS;git tag --contains 86013f47freturns exactly one tag, 0.59.0 (released 2026-08-31).0.59.0:graphistry/compute/gfql/lazy/engine/polars/hop_eager.py:118carriesuniverse = resolvable_ids.implode(), and 0.58.0 (2026-07-21) contains noimplodein that file at all.So anyone on 0.59.0 running the polars CPU engine against polars < 1.28 — which RAPIDS 25.02 pins at 1.21 via cudf-polars — hits this; 0.58.0 and earlier are unaffected. Since 0.59.0 is the current release, that combination is broken on PyPI today, so this may be worth a patch release rather than waiting for the next minor.
Also checked while confirming: 0.59.0's
hop_eager.py:557had a second membership site with a bare-Series RHS (is_in(pl.Series(drop_seed_ids, ...))). It is gone from master — unrelated work removed it — so master'shop_eagernow has exactly oneis_in, the routed one, and the changelog's "removes the remaining bare-Series deprecation warnings" claim holds.fix(gfql/polars): version-stable is_in id-set membership — polars 1.2…
test(gfql/polars): graphistry#2082 tests observe the polars deprecati…
fix(gfql/polars): graphistry#2082 review wave 1 — route gfql_unified …
fix(gfql/polars): graphistry#2082 review wave 2 — typed polars seed r…
fix(gfql/polars): graphistry#2082 review wave 3 — register the versio…
fix(gfql/polars): graphistry#2082 review wave 4 — InvalidVersion fall…
test(gfql/polars): graphistry#2082 pin both id_set arms; drop the pra…
test(gfql/polars): graphistry#2082 static lock on the is_in routing
docs(changelog): drop the internal CI count from the graphistry#2082 …