Repository navigation
CI: the test-polars py3.12 coverage cell is at its timeout ceiling; every polars test addition re-triggers a TIMEOUT reported as cancelled #1797
Description
Activity
- added a commit that references this issue
on Jul 27, 2026 The patch implementing the split is posted on #1794 (#1794 (comment)) rather than committed to that branch: the token available to the agent lacks the
workflowOAuth scope, so a commit touching.github/workflows/ci.ymlis rejected at push time. It needs a maintainer to apply it. Until thentest-polars (3.12)will keep timing out on any PR that adds polars test work.Measured evidence for follow-up 1 (
-n auto)Ran
bin/test-polars.shlocally with cuDF import-blocked (so the engine set matches the CI polars lane),-p no:randomly, same tree, back to back:workers result wall clock serial (today) 2395 passed, 54 skipped, 0 failed 172.21s -n 4(GitHub runner core count)2395 passed, 54 skipped, 0 failed 59.33s 2.9x, with an identical pass/skip set — no test-ordering or shared-state breakage at 4 workers.
-n 4rather than-n autodeliberately, to emulate a 4-core runner rather than report a wide-machine number that would not transfer.Two caveats before anyone lands this:
- Not measured under coverage.
pytest-covis not installed on the box I measured on, soPOLARS_COV=1could not be exercised. The combination is not speculative —test-gfql-corealready runspytest -n auto --cov=graphistry/computein this same workflow (ci.yml:1166) and the audit reads the resulting data file — but the polars lane's own coverage plumbing plus the per-file audit floors should be confirmed on a runner before this is relied on. - Memory. Four workers multiply peak RSS, and parts of this suite build large frames. The gfql-core precedent suggests it is fine on a 16 GB runner, but it is the failure mode to watch.
Deliberately NOT included in #1794: that PR is about variable-length query semantics, and changing the parallelism of a lane used by six python versions is a separate decision from fixing the shared-budget defect. Filed here as the evidence, not the change.
- Not measured under coverage.
The lane is a coin flip, not a fixed problem — measured
#1794 came back fully green on run
30312253135, includingtest-polars (3.12). That is not evidence the ceiling is gone. Job durations from the GitHub API, same job, near-identical workload (the PR's new test file measured +1.1s over the one it replaces):run test-polars (3.12)outcome 30304440110 attempt 2 615s cancelled(timeout at 600s)30304440110 attempt 3 611s cancelled(timeout at 600s)30312253135 431s success, 169s headroom The siblings show the same runner variance across those runs (3.11: 249s → 361s; 3.14: 351s → 329s), so the 611s → 431s move is scheduling/runner luck, not a workload change.
Concretely: this lane sits close enough to its 600s budget that the same commit can pass or time out depending on which runner it lands on. A green run must not be read as closing this issue — the fix is still the split (patch on #1794) and, longer term, follow-up 1.
Corroborating measurement, from #1805 (which widens
bin/test-polars.shby ten files):test-polars (3.12)durationbudget master 233b64c88m21s timeout-minutes: 10#1805 head 3de367109m57s timeout-minutes: 10So master is already at 84% of the budget with no change at all, and #1805 passes with a
3-second margin — green, but not meaningfully green. The other five cells are unaffected
(3.9 4m20s, 3.10 4m29s, 3.11 5m40s, 3.13 5m25s, 3.14 5m55s), which is exactly the asymmetry the
split in this issue is aimed at: one cell carries the coverage pass plus the per-file audit while
sharing a timeout sized for the plain ones.Practical consequence for sequencing: #1805 should merge after the
ci.ymlsplit lands,
otherwise the next polars test addition — or a slower-than-usual runner on the same commit —
turns this cell into acancelledwith zero test failures in the log.PR #1814 implements the xdist lever from this issue, and it is now measured ON CI rather than projected.
The
-n 4number quoted when this issue was filed (59.33s vs 172.21s, 2.9x) was measured without coverage, becausepytest-covwas absent from that environment — which was the whole risk, since 3.12 is the coverage cell. That gap is closed: coverage + xdist was verified end to end, and then the branch was run through CI.Measured,
test-polarscells, master run30300891340vs PR run30322706850:cell master PR #1814 speedup 3.12 (coverage + audit) 501s (script 484s) 322s (script 297s) 1.56x cell / 1.63x script 3.9 247s 139s 1.93x 3.10 266s 140s 2.03x 3.11 298s 145s 2.24x 3.13 340s 139s 2.63x 3.14 314s 186s 1.74x The CI speedup is smaller than the local one (a 4-CPU-pinned local A/B with coverage on gave 2.98x): coverage tracing is per-worker CPU cost that does not parallelize away, the
--cov-appendsecond phase stays serial, and 4x interpreter startup is fixed. 1.63x is the number to quote for the coverage cell.Correctness was verified rather than assumed, since a coverage regression caused by a harness bug would look like a real coverage regression in
changed-line-coverage:- node-id sets (not counts) identical serial vs parallel — 2417 node ids, 2404 passed / 13 skipped — across 4-worker, 2-worker,
--dist loadfileand 4-CPU-pinned runs - merged coverage is a strict superset of serial (28,480 vs 28,478 lines; zero lost, zero files dropped)
- the
--cov-appendphase still appends into the xdist-produced data file (+1,791 lines, nothing lost) bin/coverage_audit.py --profile gfql-polarsemits a byte-identical report from the parallel data, andcoverage combine+coverage report(whatchanged-line-coveragedoes) gives identical totals
Effect on the ordering constraint: #1805's 3.12 cell measured 597s against the 600s timeout — green by 3 seconds. At the measured 1.63x that becomes ~373s (~62% of budget), i.e. ~3.8 minutes of margin instead of 3 seconds. #1805 no longer has to wait on the
ci.ymljob split, which remains a good idea but is no longer a prerequisite that only a human withworkflowscope can unblock.- node-id sets (not counts) identical serial vs parallel — 2417 node ids, 2404 passed / 13 skipped — across 4-worker, 2-worker,
Summary
The
test-polarspy3.12 cell is at its wall-clock ceiling. It is the only cell in the matrix that runs coverage, and it shares onetimeout-minuteswith five siblings that do half its work. Adding a single test file tobin/test-polars.shis now enough to push it over, and when it goes over GitHub reports the job ascancelled— which reads as unrelated fail-fast collateral, not as a budget failure.This is a structural capacity limit, not a one-off. It will re-fire on the next polars test addition.
Evidence
On PR #1794 (which appends one file to
POLARS_TEST_FILES),test-polars (3.12)failed twice with zero test failures:cancelledcancelledJob log, attempt 3:
The pytest phase alone was 9m32s of a 10m job budget. Siblings (3.9/3.10/3.11/3.13/3.14) finish in 4m09s–5m51s. A rerun does not clear it — it is 0-for-2.
Two things made this expensive to diagnose:
cancelled, indistinguishable at a glance from fail-fast collateral. The distinguishing evidence is the duration: a fail-fast cancel lands seconds after a sibling failure; a timeout lands at exactlytimeout-minutes. Attempt 2 was 1 cancelled / 72 success / 1 skipped, so collateral was ruled out by evidence.changed-line-coveragethen reports SKIPPED, because itneeds:the job that producespolars-coverage-py3.12. So one budget overrun silently disables a second gate.Root cause
.github/workflows/ci.yml:test-polarsdeclarestimeout-minutes: 10for the whole matrix job (was line 1579).Polars tests with coverage(POLARS_COV=1, was line 1618) plus the per-filePolars GFQL coverage auditplus an artifact upload.So one cell carries roughly double the work of its siblings against a budget sized for the siblings. The timeout is simultaneously too tight for the cell that matters and too loose to be a useful hang detector for the ones that don't.
Fix applied in #1794
Split the coverage pass into its own job,
test-polars-coverage(py3.12,timeout-minutes: 20), and drop3.12from thetest-polarsmatrix. Net effect:test-polarscells now do identical work, so its 10-minute budget is a real hang detector again.changed-line-coverageneeds:updated to include the new job (it consumes that job's artifact).Note there is no
required_status_checksconfiguration onmaster(verified via the branches API), so introducing a new check name does not block merges. If required checks are added later,test-polars-coverageshould be in the list.Still open after that fix
The 20 minutes buys headroom, it does not remove the ceiling. The polars lane is:
bin/test-polars.shruns plainpython -B -m pytestwith no-n auto, unliketest-gfql-corewhich does use it. This is the single biggest available lever and is not taken.POLARS_TEST_FILESgrows monotonically, and every addition lands on the coverage lane at ~2x cost. Related: test-polars.sh is an allowlist: a new polars test file is silently never run until someone remembers to register it #1795 (new test files not in the allowlist run in no polars lane at all).Candidate follow-ups, roughly in order of value:
-n autoto the polars pytest invocations (coverage under xdist works viacoverage combine; needspytest-xdistconfirmed in thetest-polars-*locks).POLARS_TEST_FILESacross two coverage cells and combine the two.coveragefiles before the audit.Acceptance
cancelledpolars job is attributable to a real cause, not to a shared budget.