Repository navigation
fix(gfql/polars): version-stable is_in id-set membership — polars 1.21 shape errors (#2082) - #2085
Merged
Merged
Conversation
…1 shape errors (#2082) polars changed the `Expr.is_in` right-hand-side contract in 1.28.0: below it a List-typed (`implode()`) RHS is matched row-wise and must have the column's length, so the length-1 imploded universe raised `ComputeError: shapes don't match: expected N elements in 'is_in' comparison, got 1` on polars 1.21 (RAPIDS 25.02 via cudf-polars). From 1.28 the bare Series RHS is deprecated instead. One helper, `membership.is_in_ids(expr, ids)`, picks the spelling for the installed polars (`implode()` RHS >= 1.28.0, bare Series RHS below; boundary pinned by a 1.21.0..1.35.2 sweep). Routed at all 12 Series-membership sites: hop_eager (the #2082 site), chain_specializations/{hotpaths,point_rows}, pattern_apply (whose bare form also stops warning on >= 1.28, #1938 item 5). On polars >= 1.28 the emitted expression is unchanged. Tests: version predicate matrix, installed-branch invariant, set semantics with DeprecationWarning=error across int/str/categorical/float x mixed/empty/null ids against a python-list oracle, fill_null contract, LazyFrame + 50k-id universe, polars hop gate forward/reverse/undirected x int/str/cat, seeded 2-hop wavefront, empty node table. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…rding, categorical ids via Utf8, category-set canon
- polars raises its `is_in` deprecation from Rust: under a `-W error` filter it is printed,
not raised (a mutant that always used the bare Series RHS passed `-W error::DeprecationWarning`
on polars 1.35), so the membership tests record warnings and assert none is a
DeprecationWarning, the way test_hop_kernel_contracts already does.
- polars cannot cast Int64 -> Categorical directly ("cannot cast numeric types to
'Categorical'"); the categorical-id fixtures route through Utf8.
- test_seeded_node_lookup_fastpath._canon drops unused categories before comparing: the
category SET is a polars-version artifact (1.21 keeps the full rev-map through filter/join,
1.35 trims it), not part of the row contract; values and categorical dtype still compare.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…s, hoist the hop universe, lane/coverage registration - gfql_unified: the OPTIONAL arm-prune and reentry seed-restrict filters used the bare `is_in(<Series>)` form outside the engine directory; both now go through `is_in_ids` (the CHANGELOG's "remaining bare-Series warnings" claim now holds). - membership.id_set(): the prepared RHS, so hop_eager keeps master's single hoisted universe instead of imploding once per endpoint on polars >= 1.28. - bin/test-polars.sh: register the new test module (test_polars_lane_completeness gate); coverage_baselines/ci-polars-py3.12.json: baseline membership.py at 85 (one is_in_ids arm is unreachable per polars version). - tests: to_fixed_point=True so the direction matrix actually runs the eager endpoint gate (hops=1 takes the lazy semi-join lane); direct kernel pin on Utf8/Categorical ids; deprecation assertion narrowed to the polars is_in message; importorskip before engine imports; docstrings corrected. - engine_arrays.py comment dated the is_in deprecation to "polars 1.42"; it is 1.28.0. - membership docstring: why the < 1.28 branch is live despite the polars>=1.29 extra floor. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…iscriminating seeded-hop pin, doc pointers
- gfql_unified reentry seed restrict: take the ids from the polars-narrowed seed frame
(is_polars_df + is_lazy) instead of the pandas-typed SeriesT, so the call type-checks
in a polars-installed mypy env without an ignore.
- test: the seeded 2-hop pin now seeds at 2 so the result depends on the endpoint gate
({(2,3)} with it, {(2,3),(3,4)} without); renamed accordingly.
- docstrings no longer point at the gitignored plans/ directory; hotpaths comment trimmed
to the fact the name does not carry; engine_arrays comment rewrapped; coverage note
states the reproducible 93.75 figure.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…ssue-free product docstring, comment rewrap - membership._installed_polars_implodes is a maxsize=1 environment singleton; register it with cache_registry.register_process_singleton so test_clear_caches_covers_every_cache's static lru_cache scan passes. - membership.py docstring no longer cites issue numbers (bin/ci_comment_density_guard issue-rationale rule; the CHANGELOG and tests carry #2082/#1938). - engine_arrays.py comment actually rewrapped under 127 columns (wave-2 said so, the committed hunk was not). - CHANGELOG: "under RAPIDS 25.02" instead of "in that image". Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…over pragmas on the two lane-less branches - imploded_rhs_supported falls back to the bare-Series RHS on an unparseable version string instead of raising: that RHS is correct on every release 1.21..1.35 and only warns from 1.28, so it is the safe side (idiom: plugins/networkx/policy.py). Test rows for local (+cu12), dev (both sides of the floor), unparseable and empty versions. - Two changed branches have no CI lane and so cannot be covered by bin/changed_line_coverage.py: the polars arm of _optional_arm_start_nodes (unreachable — its caller routes polars engines to _optional_arm_membership_chain) and id_set's polars < 1.28 arm (every CI lane pins polars >= 1.29; validated on dgx RAPIDS 25.02). Both now carry the repo's `# pragma: no cover - <reason>` idiom, which also records the unreachability in the code rather than only in review notes. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…n replaces The polars < 1.28 arm is the whole point of the fix and no CI lane installs a polars that old, so it had neither a lane nor a test: collapsing id_set to a single spelling would have stayed green everywhere and silently brought the RAPIDS 25.02 `ComputeError: shapes don't match ... 'is_in'` back. A monkeypatched pin now asserts both spellings, which also covers the line, so its `# pragma: no cover` comes off. Also: unparseable-version fallback split into its own named test; the coverage baseline records the measured figure and a floor with the file's customary margin; and one docstring covers both dead polars arms of _optional_arm_start_nodes instead of pragma-ing one and leaving its sibling unmarked. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
The value-level pins protect the helper, not the call sites. A future `pl.col(x).is_in(ids.implode())` written directly is correct and warning-free on every CI polars (>= 1.29) and raises ComputeError on the polars 1.21 that RAPIDS 25.02 pins, so no value test in any lane can catch it. This AST scan does. What it catches: the imploded RHS written in the call, by argument or keyword, and through a name bound by a plain, annotated or walrus assignment in the same module — the spelling the reported site itself used (`universe = resolvable_ids.implode()` then `.is_in(universe)`). The bound value is walked, so an implode wrapped in another call still binds the name; that over-approximates, which fails loudly rather than silently. Not caught, and said so in the comment: binding through a tuple unpack, an attribute or a helper's return value, which needs real dataflow, and a bare-Series RHS, which is undecidable statically and only warns. The lock carries its own controls, because one that silently scans nothing would pass forever: the exempt helper must exist and sit inside the walked tree, four routed modules must be among the files parsed (including gfql_unified.py, outside the engine subtree, so a re-narrowing cannot keep every other pin green), the file count must clear a floor far below the ~350 shipped modules, and the matcher is exercised against 16 spellings — 7 it must flag, 9 it must not. Three mutations of hop_eager back to the pre-fix spelling (plain, annotated, wrapped) each flag line 122; the tree scans clean. Coverage baseline: membership.py at 92.0, ~8% below the 100.0 measured on the dgx polars lane, in the file's customary note form. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
"1792 of the 2361 polars-engine test failures" is a number for the pull request, not for a user-facing changelog; the entry is already the longest in the section. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…cosmetic Every existing pin exercises the arm the installed polars selects, so nothing asserted that the OTHER arm is actually wrong — the gate could have been decorative and the suite would not have noticed. Only a throwaway mutant showed otherwise, and that is not in the repo. This forces the imploded arm and asserts the consequence, version-agnostically: below 1.28 it must raise the #2082 ComputeError, which is the whole reason the gate exists; from 1.28 both spellings are legal, so the forced arm must agree with the routed one value for value. On a RAPIDS 25.02 box that is the raise branch, in CI it is the equivalence branch, and neither lane skips. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2082: on polars 1.21 (the version RAPIDS 25.02 pins through cudf-polars) the native polars GFQL engine raised
wherever it tested ids against a Series via
is_in(ids.implode()). polars changed theis_inright-hand-side contract in 1.28.0: below it a List-typed RHS is matched row-wise and must have the column's length; from 1.28 the bare-Series RHS is deprecated instead (pola-rs/polars#22149), which is why.implode()was introduced on 2026-08-15 (#1895 review) and why 1.21 broke.One helper,
graphistry/compute/gfql/lazy/engine/polars/membership.py, picks the spelling for the installed polars:id_set(ids)returns the prepared RHS (implode()on >= 1.28.0, the flat Series below);is_in_ids(expr, ids)wraps it. All 14 Series-membership sites go through it:hop_eager(the reported site),chain_specializations/{hotpaths,point_rows},pattern_apply, and two sites ingfql_unified(OPTIONAL arm prune, reentry seed restrict) that still used the bare form and therefore still warned on >= 1.28 (#1938 item 5). On polars >= 1.28 the emitted expressions are unchanged, including the hoisted hop universe.Version boundary
Pinned by a sweep over 14 polars releases 1.21.0 … 1.35.2 (venv per version, same 8-scenario probe: int/str/categorical × nonempty/empty/null-in-universe): the imploded RHS fails on every release through 1.27.1 and passes from 1.28.0; the bare RHS passes on every release and warns from 1.28.0.
IMPLODED_RHS_FLOOR = Version("1.28.0").The
polarsextra declarespolars>=1.29, so the< 1.28branch serves only environments pinned below the extra's floor by another package — today RAPIDS 25.02. The module docstring says so, so nobody deletes the branch as dead.Evidence (DGX Spark, exact trees shipped by
git archive, sha256 manifests)25.02-gfql(1.21.0)tests/compute/gfql -k polars: 2361 failed (1792 = this idiom)polars-gpuparams that cudf-polars 25.02 cannot execute, 0 polars-CPU failures26.02-gfql(1.35.2)26.02-gfql-polars,TEST_POLARS_GPU=1(1.35.2)Static gates on this exact tree: ruff clean (both passes),
bin/ci_type_hygiene_guard.pyandbin/ci_comment_density_guard.pyexit 0 with no growth, whole-tree mypy identical to master (258 pre-existing errors on both, 0 new). The polars coverage lane pluscoverage_audit --profile gfql-polarswas replicated too:membership.pymeasures 100% against its 92.0 floor.Controls: a mutant that always implodes re-breaks 100 tests on 1.21; a mutant that always uses the bare form is caught by the record-and-assert warning pins on 1.35 (polars emits that deprecation from Rust and prints it under
-W error, so-W erroralone is blind — the tests record instead).The 499 remaining failures on polars 1.21 are out of scope and pre-existing: 365
polars-gpuNotImplementedError / 54 GFQLUnsupportedError (cudf-polars 25.02 cannot execute those plans; the 25.02 GPU lane is not claimed), 3Expr.round(mode=)(polars >= 1.29 feature, the extra's floor), 4 row-op NotImplementedError, 4 harness artifacts.Tests
test_polars_membership_isin.py(registered inbin/test-polars.sh): version-predicate matrix including local, dev and pre-release strings and an unparseable fallback; bothid_setarms pinned regardless of the installed polars, so collapsing it to one spelling fails; set semantics with recorded-warning assertions across int/str/categorical/float x mixed/empty/null-in-universe against a python-list oracle and hand literals;fill_nullcontract; LazyFrame with a 50k-id universe; the polars hop endpoint gate forward/reverse/undirected x int/str/categorical underto_fixed_pointso the eager gate is what runs; a seeded 2-hop whose answer depends on the gate; empty node table; direct kernel pin on Utf8/Categorical ids.pl.col(x).is_in(ids.implode())written directly, because it is correct and warning-free on every CI polars and fatal on RAPIDS 25.02. An AST scan over the ~350 shipped modules fails on that spelling outside the helper, following names bound by plain, annotated or walrus assignment and walking the bound value. It does not chase tuple unpacks, attributes, cross-function returns, or a bare-Series RHS, and says so. It carries its own controls: the exempt helper must exist inside the walked tree, four routed modules must be among the files parsed, a file-count floor, and a 16-case matcher matrix. Three mutations ofhop_eagerback to the pre-fix spelling each trip it.test_seeded_node_lookup_fastpath._canondrops unused categories before comparing: the category set is a polars-version artifact (1.21 keeps the full rev-map through filter/join, 1.35 trims it), not part of the row contract; values and categorical dtype still compare exactly. Without this, 2 of the 33 cases pass theis_infix and then fail on category-set parity on 1.21 only.membership.pybaselined incoverage_baselines/ci-polars-py3.12.jsonat 92.0, against a measured 100%.Limitations
polarsextra floor (1.29); features gated on >= 1.29 (round(mode=)) still fail there by design.gfql_unified._optional_arm_start_nodes' polars arms are unreachable today (the caller sends polars engines to_optional_arm_membership_chain) and were equally dead before this change. They are kept because a routing change would otherwise reach the pandas.isinwith polars frames; the docstring and a# pragma: no coversay so.lazy/engine/polars/dtypes.pymeasures 87.30% against its 92.0 floor. Unmodified master produces the identical number in the same image, and this branch does not touch that file, so it is environmental (the image is py3.13 with cuDF; the CI lane is py3.12 without). No baseline was changed for it.Plan, per-version sweep JSON and raw lane logs: local
plans/gfql-2082-polars121/(gitignored).🤖 Generated with Claude Code
https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud