Skip to content

fix(gfql): type the polars chain combine helpers; TypeIs for is_lazy instead of a cast - #1789

Merged
lmeyerov merged 1 commit into
masterfrom
type/polars-chain-typeis
Jul 27, 2026
Merged

lmeyerov merged 1 commit into
masterfrom
type/polars-chain-typeis

Conversation

@lmeyerov

Copy link
Copy Markdown
Contributor

Four helpers in the native polars chain engine were untyped, and one of them papered over the resulting hole with a cast. This replaces all of them with real types — no Any, cast, getattr, setattr, list, Dict[str, Any], or object was used to get there.

The cast, and why TypeIs is the fix rather than a workaround

dtypes.is_lazy answers which member of the two-member PolarsFrame union a frame is, but it was declared -> bool, so its answer was thrown away at the call boundary. That made _known_empty unprovable:

if frame is None or is_lazy(frame):
    return None
return frame.height == 0        # `frame` is still DataFrame|LazyFrame to mypy

The old code closed the gap with cast("pl.DataFrame", frame). A cast is not a type. It is an unchecked re-assertion of the exact fact the predicate had just established and then discarded, it has to be repeated at every eager-side call site (each an independent chance to be wrong), and it is silently wrong: a cast never fails at runtime, so a mistake surfaces as an AttributeError in production instead of an error in CI.

Widening the predicate to -> TypeIs["pl.LazyFrame"] (PEP 742) makes the fact survive the call, so .height type-checks with no assertion at all. TypeIs, not TypeGuard: TypeGuard narrows only the positive branch, and here every eager-side caller sits in the else — narrowing the negative branch is the entire point. Because the fix is at the definition, it holds for every is_lazy caller in the engine (chain.py, degrees.py, pattern_apply.py), not just the one site that had the cast.

The import is TYPE_CHECKING-only and dtypes.py is from __future__ import annotations, so no runtime typing_extensions floor is introduced, and is_lazy's body is byte-identical.

The other three

helper was now
_semi (df, ids_df, df_col, id_col) (df: PolarsT, ids_df: PolarsT, df_col: str, id_col: str) -> PolarsT
_combine_edges (g, steps, label_steps, has_multihop=False) (g: _LazyShim, steps: List[Tuple[ASTObject, _LazyShim]], label_steps: List[Tuple[ASTObject, _LazyShim]], has_multihop: bool = False) -> pl.LazyFrame
_apply_node_names (out, g, steps, auto_hop_col=...) (out: pl.LazyFrame, g: _LazyShim, steps: List[Tuple[ASTObject, _LazyShim]], auto_hop_col: str = ...) -> pl.LazyFrame
  • _semi's two frames share the same PolarsT TypeVar, not PolarsFrame on each: polars joins do not mix eagerness (DataFrame.join takes a DataFrame, LazyFrame.join takes a LazyFrame, a mixed pair raises), and a semi-join preserves the left frame's eagerness. Same variable in, same variable out is the true signature; a union would be both less true and unusable downstream.
  • The two combine helpers take the _LazyShim collect-once duck-type, not Plottable — the Track-B combine runs entirely on already-materialized-then-lazified hop frames. This matches the already-correctly-typed _combine_node_ids / _materialize_node_rows next to them.

Two escape hatches fell out as a consequence of typing rather than being worked around:

  • getattr(next_step, "edges_empty", None) → next_step.edges_empty. _LazyShim declares edges_empty in __slots__ with a real Optional[bool], so the tri-state is now part of the type and a typo is a checker error rather than a silent None — which would have re-armed the exact cardinality gate that guard exists to disarm.
  • g._edges is Optional[pl.LazyFrame], previously dereferenced twice on faith; it is now discharged once with an assert at the top of _combine_edges.

Not fixed here (stated, not smuggled)

_exec's prev_wf / target_wf / intermediate_universe are still Optional[Any]. They are pre-existing, outside the four helpers this PR was scoped to, and typing them means typing hop_lazy_or_eager's wavefront parameters too — a separate change with a separate blast radius. Flagging rather than hiding it.

Verification

All on dgx-spark, in-container (graphistry/test-rapids-official:26.02-gfql-polars, mypy 1.19.1, ruff 0.15.8, polars 1.35.2), against an unmodified 3f2128ea clone in the same image.

mypy differential — the absolute count is meaningless here (this image's pandas-stubs drift from the CI lockfile); only the differential is evidence:

master 3f2128ea this branch
mypy graphistry/ 173 errors / 324 files 173 errors / 324 files
error sets (file:code:message) — byte-identical diff
mypy chain.py alone Success: no issues found Success: no issues found

ruff (repo pyproject.toml): ruff check graphistry/ → All checks passed!

Test suite — graphistry/tests/compute, --gpus all (required: without it the suite reports failures rather than skips):

result
master 3f2128ea 9 failed, 7020 passed, 90 skipped, 15 xfailed in 197.47s
this branch 9 failed, 7020 passed, 90 skipped, 15 xfailed in 203.77s
failure sets identical (diff clean) — the 9 are the pre-existing dask / cuDF-igraph coercion failures

Runtime behaviour is unchanged by construction: is_lazy's body is untouched, the TypeIs import never executes, and every other edit is an annotation, an assert discharging an already-assumed Optional, or the getattr → attribute swap.

🤖 Generated with Claude Code

https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB

…instead of a cast

Four helpers in the native polars chain engine were untyped, and one of them papered
over the resulting hole with `cast("pl.DataFrame", frame)`. Replace all five with real
types.

Why TypeIs and not a cast
-------------------------
`dtypes.is_lazy` answers WHICH member of the two-member `PolarsFrame` union a frame is,
but it was declared `-> bool`, so its answer was thrown away at the boundary. That made
`_known_empty` unprovable:

    if frame is None or is_lazy(frame):
        return None
    return frame.height == 0        # frame is still DataFrame|LazyFrame to mypy

The old code closed the gap with `cast("pl.DataFrame", frame)`. A cast is not a type — it
is an unchecked assertion of the exact fact the predicate had just established and then
discarded, and it has to be re-asserted at every eager-side call site, each one an
independent chance to be wrong (and silently wrong: a cast never fails at runtime, so a
mistake surfaces as an AttributeError in production rather than an error in CI).

Widening the predicate to `-> TypeIs["pl.LazyFrame"]` (PEP 742) makes the fact survive the
call, so `.height` type-checks with no assertion at all. TypeIs, not TypeGuard: TypeGuard
narrows only the positive branch, and here every eager-side caller sits in the `else`.
The fix is at the definition, so it holds for every `is_lazy` caller in the engine
(chain.py, degrees.py, pattern_apply.py) rather than at one site.

The import is TYPE_CHECKING-only and dtypes.py is `from __future__ import annotations`,
so no runtime typing_extensions floor is introduced and `is_lazy`'s body is untouched.

The other three
---------------
- `_semi(df, ids_df, df_col, id_col)` -> both frames share the `PolarsT` TypeVar. polars
  joins do not mix eagerness (`DataFrame.join` takes a DataFrame, `LazyFrame.join` takes a
  LazyFrame; a mixed pair raises), and a semi-join preserves the left frame's eagerness —
  so same variable in, same variable out is the true signature, not a union.
- `_combine_edges` / `_apply_node_names` -> `_LazyShim` and
  `List[Tuple[ASTObject, "_LazyShim"]]`, matching the already-typed `_combine_node_ids`.
  These take the Track-B shim, NOT a `Plottable`: the combine runs entirely on
  already-materialized-then-lazified hop frames.

Typing `_apply_node_names` also retires a `getattr(next_step, "edges_empty", None)` in
favour of a plain attribute read: `_LazyShim` declares `edges_empty` in `__slots__` with a
real `Optional[bool]`, so the tri-state is now part of the type and a typo is a checker
error instead of a silent `None` — which would have re-armed the exact gate that guard
disarms. Typing `_combine_edges` likewise forced `g._edges`' `Optional` to be discharged
once by assert instead of being implicitly assumed at two use sites.

No `Any`, `cast`, `getattr`, `object` or `Dict[str, Any]` was used to reach this; nothing
was left untyped by falling back to one.

Verification (dgx-spark, graphistry/test-rapids-official:26.02-gfql-polars)
--------------------------------------------------------------------------
- mypy differential vs an unmodified 3f2128e tree, same container, same config:
  173 errors before / 173 after over `graphistry/`, error sets byte-identical
  (the 173 are pre-existing pandas-stubs drift in this image, not from the CI lockfile;
  only the differential is evidence). `mypy chain.py` alone: clean on both.
- `ruff check graphistry/`: clean.
- `graphistry/tests/compute` with `--gpus all`: failure set identical to master's.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant