Repository navigation
fix(gfql): type the polars chain combine helpers; TypeIs for is_lazy instead of a cast - #1789
Merged
Merged
Conversation
…instead of a cast
Four helpers in the native polars chain engine were untyped, and one of them papered
over the resulting hole with `cast("pl.DataFrame", frame)`. Replace all five with real
types.
Why TypeIs and not a cast
-------------------------
`dtypes.is_lazy` answers WHICH member of the two-member `PolarsFrame` union a frame is,
but it was declared `-> bool`, so its answer was thrown away at the boundary. That made
`_known_empty` unprovable:
if frame is None or is_lazy(frame):
return None
return frame.height == 0 # frame is still DataFrame|LazyFrame to mypy
The old code closed the gap with `cast("pl.DataFrame", frame)`. A cast is not a type — it
is an unchecked assertion of the exact fact the predicate had just established and then
discarded, and it has to be re-asserted at every eager-side call site, each one an
independent chance to be wrong (and silently wrong: a cast never fails at runtime, so a
mistake surfaces as an AttributeError in production rather than an error in CI).
Widening the predicate to `-> TypeIs["pl.LazyFrame"]` (PEP 742) makes the fact survive the
call, so `.height` type-checks with no assertion at all. TypeIs, not TypeGuard: TypeGuard
narrows only the positive branch, and here every eager-side caller sits in the `else`.
The fix is at the definition, so it holds for every `is_lazy` caller in the engine
(chain.py, degrees.py, pattern_apply.py) rather than at one site.
The import is TYPE_CHECKING-only and dtypes.py is `from __future__ import annotations`,
so no runtime typing_extensions floor is introduced and `is_lazy`'s body is untouched.
The other three
---------------
- `_semi(df, ids_df, df_col, id_col)` -> both frames share the `PolarsT` TypeVar. polars
joins do not mix eagerness (`DataFrame.join` takes a DataFrame, `LazyFrame.join` takes a
LazyFrame; a mixed pair raises), and a semi-join preserves the left frame's eagerness —
so same variable in, same variable out is the true signature, not a union.
- `_combine_edges` / `_apply_node_names` -> `_LazyShim` and
`List[Tuple[ASTObject, "_LazyShim"]]`, matching the already-typed `_combine_node_ids`.
These take the Track-B shim, NOT a `Plottable`: the combine runs entirely on
already-materialized-then-lazified hop frames.
Typing `_apply_node_names` also retires a `getattr(next_step, "edges_empty", None)` in
favour of a plain attribute read: `_LazyShim` declares `edges_empty` in `__slots__` with a
real `Optional[bool]`, so the tri-state is now part of the type and a typo is a checker
error instead of a silent `None` — which would have re-armed the exact gate that guard
disarms. Typing `_combine_edges` likewise forced `g._edges`' `Optional` to be discharged
once by assert instead of being implicitly assumed at two use sites.
No `Any`, `cast`, `getattr`, `object` or `Dict[str, Any]` was used to reach this; nothing
was left untyped by falling back to one.
Verification (dgx-spark, graphistry/test-rapids-official:26.02-gfql-polars)
--------------------------------------------------------------------------
- mypy differential vs an unmodified 3f2128e tree, same container, same config:
173 errors before / 173 after over `graphistry/`, error sets byte-identical
(the 173 are pre-existing pandas-stubs drift in this image, not from the CI lockfile;
only the differential is evidence). `mypy chain.py` alone: clean on both.
- `ruff check graphistry/`: clean.
- `graphistry/tests/compute` with `--gpus all`: failure set identical to master's.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four helpers in the native polars chain engine were untyped, and one of them papered over the resulting hole with a
cast. This replaces all of them with real types — noAny,cast,getattr,setattr,list,Dict[str, Any], orobjectwas used to get there.The cast, and why
TypeIsis the fix rather than a workarounddtypes.is_lazyanswers which member of the two-memberPolarsFrameunion a frame is, but it was declared-> bool, so its answer was thrown away at the call boundary. That made_known_emptyunprovable:The old code closed the gap with
cast("pl.DataFrame", frame). A cast is not a type. It is an unchecked re-assertion of the exact fact the predicate had just established and then discarded, it has to be repeated at every eager-side call site (each an independent chance to be wrong), and it is silently wrong: a cast never fails at runtime, so a mistake surfaces as anAttributeErrorin production instead of an error in CI.Widening the predicate to
-> TypeIs["pl.LazyFrame"](PEP 742) makes the fact survive the call, so.heighttype-checks with no assertion at all.TypeIs, notTypeGuard:TypeGuardnarrows only the positive branch, and here every eager-side caller sits in theelse— narrowing the negative branch is the entire point. Because the fix is at the definition, it holds for everyis_lazycaller in the engine (chain.py,degrees.py,pattern_apply.py), not just the one site that had the cast.The import is
TYPE_CHECKING-only anddtypes.pyisfrom __future__ import annotations, so no runtimetyping_extensionsfloor is introduced, andis_lazy's body is byte-identical.The other three
_semi(df, ids_df, df_col, id_col)(df: PolarsT, ids_df: PolarsT, df_col: str, id_col: str) -> PolarsT_combine_edges(g, steps, label_steps, has_multihop=False)(g: _LazyShim, steps: List[Tuple[ASTObject, _LazyShim]], label_steps: List[Tuple[ASTObject, _LazyShim]], has_multihop: bool = False) -> pl.LazyFrame_apply_node_names(out, g, steps, auto_hop_col=...)(out: pl.LazyFrame, g: _LazyShim, steps: List[Tuple[ASTObject, _LazyShim]], auto_hop_col: str = ...) -> pl.LazyFrame_semi's two frames share the samePolarsTTypeVar, notPolarsFrameon each: polars joins do not mix eagerness (DataFrame.jointakes aDataFrame,LazyFrame.jointakes aLazyFrame, a mixed pair raises), and a semi-join preserves the left frame's eagerness. Same variable in, same variable out is the true signature; a union would be both less true and unusable downstream._LazyShimcollect-once duck-type, notPlottable— the Track-B combine runs entirely on already-materialized-then-lazified hop frames. This matches the already-correctly-typed_combine_node_ids/_materialize_node_rowsnext to them.Two escape hatches fell out as a consequence of typing rather than being worked around:
getattr(next_step, "edges_empty", None)→next_step.edges_empty._LazyShimdeclaresedges_emptyin__slots__with a realOptional[bool], so the tri-state is now part of the type and a typo is a checker error rather than a silentNone— which would have re-armed the exact cardinality gate that guard exists to disarm.g._edgesisOptional[pl.LazyFrame], previously dereferenced twice on faith; it is now discharged once with anassertat the top of_combine_edges.Not fixed here (stated, not smuggled)
_exec'sprev_wf/target_wf/intermediate_universeare stillOptional[Any]. They are pre-existing, outside the four helpers this PR was scoped to, and typing them means typinghop_lazy_or_eager's wavefront parameters too — a separate change with a separate blast radius. Flagging rather than hiding it.Verification
All on dgx-spark, in-container (
graphistry/test-rapids-official:26.02-gfql-polars, mypy 1.19.1, ruff 0.15.8, polars 1.35.2), against an unmodified3f2128eaclone in the same image.mypy differential — the absolute count is meaningless here (this image's pandas-stubs drift from the CI lockfile); only the differential is evidence:
3f2128eamypy graphistry/file:code:message)diffmypy chain.pyaloneSuccess: no issues foundSuccess: no issues foundruff (repo
pyproject.toml):ruff check graphistry/→All checks passed!Test suite —
graphistry/tests/compute,--gpus all(required: without it the suite reports failures rather than skips):3f2128ea9 failed, 7020 passed, 90 skipped, 15 xfailed in 197.47s9 failed, 7020 passed, 90 skipped, 15 xfailed in 203.77sdiffclean) — the 9 are the pre-existing dask / cuDF-igraph coercion failuresRuntime behaviour is unchanged by construction:
is_lazy's body is untouched, theTypeIsimport never executes, and every other edit is an annotation, anassertdischarging an already-assumedOptional, or thegetattr→ attribute swap.🤖 Generated with Claude Code
https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB