Typing: 173 mypy errors visible under newer pandas-stubs that CI's pinned lockfile does not see #1791

Activity

1. lmeyerov commented on Jul 27, 2026

   

   lmeyerov

   on Jul 27, 2026

   ContributorAuthor

   More actions

   Triage — and the premise is wrong in an interesting way

   Short version: this is not a stub-pin problem. CI's type gate is switched off.

   CI's own pinned stubs already see 140 errors

   environment	mypy	pandas-stubs	result
   CI, as it actually runs today	2.3.0 (uvx)	none installed	Success: no issues found in 324 source files
   CI's own locked env (requirements/test-py3.12.lock, run #30287083453 on f89e6378)	2.1.0	3.0.3.260530	140 errors in 10 files
   test-rapids-official:26.02-gfql-polars	1.19.1	3.0.0.260204	173 errors in 12 files

   CI's pandas-stubs is newer than the container's (Mar 30 vs Feb 4 build), and it reports fewer errors. Bumping the pin is not the lever.

   Root cause: bin/typecheck.sh prefers uvx mypy

   if command -v uvx >/dev/null 2>&1; then
     MYPY_CMD="uvx mypy"

   CI's install step is python -m pip install --upgrade pip uv, which puts uvx on PATH. uvx runs mypy in an isolated ephemeral environment, so the uv pip install --require-hashes -r requirements/test-py3.12.lock install of pandas-stubs / types-requests / types-tqdm / types-defusedxml, and uv pip install -e . of the project's own deps, are all invisible to it. mypy.ini then has ignore_missing_imports = True for pandas.*, numpy.*, pyarrow.*, requests.*, … so every external symbol degrades to Any and 324 files pass vacuously.

   Two independent confirmations:

   1. The CI log installs mypy==2.1.0 from the lockfile, then the run prints mypy 2.3.0 (compiled: yes). Different interpreter, different mypy.
   2. mypy --no-site-packages reproduces CI's output byte for byte, including the file count: Success: no issues found in 324 source files.

   bin/lint.sh has the same uvx ruff shape. Much less harmful — ruff needs no imports and still reads pyproject.toml — but the locked ruff version is likewise bypassed, so a new ruff release can turn CI red with no repo change, and a lockfile ruff bump cannot fix it.

   cypher-frontend-strict-typing calls bare mypy (the venv's, with stubs), so that job is a genuine gate — but only over gfql/ir/*.py + cypher/binder.py with --follow-imports=skip.

   ---

   Triage table — 140 errors under CI's own pin

   Pattern count, not error count. Four patterns explain 137 of 140.

   #	cluster	bucket	count	evidence
   P1	polars API called on a DataFrameT-annotated value	not a stub artifact — the repo's alias is the lie	~124	see below
   P2	hand-rolled ArrayLike Protocol vs concrete numpy	modelling gap (neither stub nor code bug)	6	traverse.py:165,167,193,195,209, engine_arrays.py:69
   B1	search_any.py: pat bound to both a module and a str in one scope	B — real defect	6	genuine shadowed name
   B2	graphviz.py: row[g._node] where _node is Optional[str]	B — real defect, reproducible	3	KeyError: None
   C1	search_any.py params typed dtype: object	annotation correctness (object is banned here)	3	:25, :30
   A1	join.py:102,105 — DataFrame.take(Series)	A — genuine stub artifact	2	works at runtime, stub is stricter
   C2	cast()-induced declared-type narrowing	symptom of the banned cast	1	reentry/execution.py:349

   P1 — the 124-error mega-cluster (88%)

   compute/typing.py declares, under TYPE_CHECKING only:

   DataFrameT = pd.DataFrame

   …as a stand-in for "whichever engine's frame". Every polars-only fast path is annotated DataFrameT but receives a pl.DataFrame. Under stubs that actually model pandas, each polars call is an error. It decomposes into exactly three shapes:

   shape	count	why
   "Series[Any]" not callable [operator]	49	.select ×20, .unique ×11, .len ×5, .group_by ×5, .cast ×3, .lazy ×2, .with_columns, .join. pandas-stubs types DataFrame.__getattr__ -> Series[Any] for attribute-style column access, so any polars method name resolves to a Series and "calling" it errors. "bool" not callable (traverse.py:163,191) is the same thing one type deeper: (s == v) → Series[bool], .fill_null → bool.
   Unexpected keyword argument "left_on"/"right_on" for "join" [call-arg]	36	polars is join(other, left_on=, right_on=); pandas join takes on= only (left_on/right_on are merge).
   Argument "how" ... "Literal['semi']" [arg-type]	27	how="semi" is polars.

   The stub is right and the annotation is wrong — the opposite of the guess in the issue body. pandas DataFrame.join genuinely has no left_on; how="semi" genuinely does not exist in pandas or cuDF (cuDF spells it "leftsemi" on merge). And every offending call site is inside a polars gate: if engine in POLARS_ENGINES, if engine in (Engine.POLARS, Engine.POLARS_GPU), or a function carrying # pragma: no cover - polars-only, covered by polars lane.

   So P1 is neither a stub bug nor a runtime bug — it is DataFrameT = pd.DataFrame used where the value is provably polars. The fix is the real refactor: type the polars-only functions with pl.DataFrame / the PolarsFrame / PolarsT aliases from lazy/engine/polars/dtypes.py, exactly as #1789 did for the chain combine helpers. That also deletes the ~30 existing cast(DataFrameT, ...) calls in these files, which are on the banned list anyway. Not attempted here — it is a large, mechanical, separately-reviewable change.

   A related corroboration that error-code-pinned suppression rots: Engine.py:920 and :930 already carry # type: ignore[attr-defined,no-any-return], written against some earlier stub set. Under real stubs the emitted code is [operator], so those ignores are silently dead and the errors come through anyway.

   B1 — search_any.py, real shadowed name (6 errors)

           import pandas.api.types as pat          # line 95   -> module
           ...
                       or bool(pat.is_bool_dtype(dt))):   # line 100  -> module
       ...
       pat, case = term, case_sensitive            # line 110  -> str
       pred = Contains(pat, case=case, ...)        # line 118

   Exactly the shape predicted in the issue. Ordering saves it today — the module use strictly precedes the rebind — so it is a latent hazard, not a live failure, and I am not going to overclaim it. But it is one statement reorder (or one new dtype check after line 110) from AttributeError: 'str' object has no attribute 'is_bool_dtype'. Fixed in #1792 by renaming the alias to pd_types. No behaviour change, so no test; the mypy differential is the evidence.

   B2 — graphviz.py, real reproducible defect (3 errors)

   g_to_pgv asserts _nodes/_edges are set, but not that the bindings are. g.nodes(df) with no node= leaves _node None while _nodes is set — and layout_graphviz/render_graphviz only call materialize_nodes() when _nodes is None. So that graph sails through to row[None]:

   g = graphistry.edges(edf, 's', 'd').nodes(pd.DataFrame({'id': ['a','b']}))
   g._node = None
   g_to_pgv(g)  ->  KeyError : None
   

   Same for g.edges(df) with no source/destination (_source/_destination at :78,:79). Fixed in #1792 with an actionable ValueError, plus two regression tests that fail on master.

   A1 — the one genuine stub artifact

   join.py:102,105: Argument 1 to "take" of "NDFrame" has incompatible type "Series[Any]". Verified at runtime on pandas 2.3.3 — df.take(pd.Series([2,0])) returns [30, 10] fine. Stub is stricter than pandas. Two errors. This is the only cluster where "the stub is wrong" is the correct answer.

   ---

   Recommendation on the CI stub pin

   Do not bump pandas-stubs. Stop bypassing it.

   setup.py has stubs = ['pandas-stubs', ...] unpinned, resolved per-Python-version by bin/generate-lockfiles.sh under a 6-day --exclude-newer cooldown. That is a reasonable policy and it already lands on a current stub set. The pin is not the problem; nothing reads it.

   Suggested sequencing, since a straight fix flips CI from green to 128 errors:

   1. Make the feedback visible first. Add a typecheck-with-stubs job that runs the venv's mypy (python -m mypy --config-file mypy.ini graphistry/) against the same locked env, with continue-on-error: true. Zero risk, and the 128 stop being invisible today.
   2. Drive it to zero, P1 first (~124 of 128 — one mechanical refactor across gfql_fast_paths.py + chain_fast_paths.py, retyping polars-only helpers to PolarsFrame/PolarsT and deleting the cast(DataFrameT, ...) calls on the way).
   3. Then fix the resolution order in bin/typecheck.sh — prefer the active interpreter's mypy, fall back to uvx only when none is installed — and drop continue-on-error. Keep uvx as the fallback so local contributors without a venv still get a check; just never let it silently outrank the locked one.
   4. Same reordering for bin/lint.sh, so the locked ruff is what runs. Lower priority, but it is the same class of bug and it makes CI non-reproducible in the other direction.
   5. Optional hardening: warn_unused_ignores = True in mypy.ini once (2) lands, which would have caught the dead Engine.py:920 ignores.

   One more thing worth knowing about the lockfile job: its cache key is hashFiles('setup.py', 'pyproject.toml', 'requirements/*.txt', 'requirements/*.in', 'bin/generate-lockfiles.sh'). Those change rarely, so the 6-day cooldown does not mean "stubs are at most 6 days old" — it means "at most 6 days older than the last time one of those files changed". Not a problem today, but it means the effective stub age drifts on an unrelated schedule.

   ---

   PR

   #1792 fixes B1 + B2 + C1 — 12 of the 140, zero suppressions, zero Any/cast/getattr/setattr/object. mypy 140 → 128 under CI's own pin with zero new errors (error-set diff), ruff clean, graphistry/tests/compute --gpus all at 9 failed / 7030 passed, identical to the master baseline.

   P1 (124) and P2 (6) are deliberately left — they want the real refactor described above, not suppression.

   Reproducing any of this

   docker run --rm --entrypoint bash \
     -v <tree>:/opt/pygraphistry -e PYTHONPATH=/opt/pygraphistry -e PYTHONDONTWRITEBYTECODE=1 \
     -w /opt/pygraphistry graphistry/test-rapids-official:26.02-gfql-polars -lc '
       python3 -m venv /tmp/v
       /tmp/v/bin/pip -q install mypy==2.1.0 pandas-stubs==3.0.3.260530 \
           types-requests types-defusedxml types-tqdm
       echo "--- CI\x27s locked env ---"
       /tmp/v/bin/mypy --cache-dir=/tmp/mc1 --config-file mypy.ini graphistry/ | tail -1
       echo "--- what CI actually runs (uvx isolation) ---"
       /tmp/v/bin/mypy --no-site-packages --cache-dir=/tmp/mc2 --config-file mypy.ini graphistry/ | tail -1
     '

   --- CI's locked env ---
   Found 140 errors in 10 files (checked 324 source files)
   --- what CI actually runs (uvx isolation) ---
   Success: no issues found in 324 source files
   

   🤖 Generated with Claude Code
2. lmeyerov

   added a commit that references this issue on Jul 27, 2026

   fix(typing): two real defects that CI's type gate cannot see (#1791) (#…

   ...

   c76f23a