Skip to content

Repair test-report tooling - #9321

Merged
crusaderky merged 11 commits into
dask:mainfrom
crusaderky:test-report
Jul 9, 2026
Merged

crusaderky merged 11 commits into
dask:mainfrom
crusaderky:test-report

Conversation

@crusaderky

@crusaderky crusaderky commented Jul 3, 2026 •

Copy link
Copy Markdown
Collaborator

The Test Report workflow (runs) has been failing for a very long time. This PR resurrects it and migrates it to pixi.

New feature: gh CLI fallback

When GITHUB_TOKEN is not set, the token is read from gh auth token. This makes it easier to run the tool locally.

Follow-ups

  • A follow-up PR in dask/dask migrates its test-report workflow to use this same tooling.
  • The artifact databases grow unboundedly over time (~25 MB/day pre-compression for dask/distributed); entries for runs that fall out of the 30-day window are never pruned. Since GitHub evicts caches LRU at 10 GB per repo this is tolerable, but a pruning pass would be a good follow-up.
  • The local database system is slow and overcomplicated; it could use simplification.

The Test Report workflow has been broken since the CI migration to pixi (dask#9276):

- Run the report generation through a new, self-contained `test-report` pixi
  environment: `pixi run test-report dask/distributed`.
  Delete the unused conda environment file.
- Adapt job/artifact name parsing in test_report.py to the names produced by
  the post-pixi tests.yaml, in both dask/distributed and dask/dask. Runs older
  than 2026-06-04 cannot be parsed and are skipped.
- Replace the unmaintained altair_saver with altair>=6 native HTML output.
- Fix regression introduced by the f-string conversion in dask#9245 which
  collapsed all tests into a single unreadable chart.
- Name the local artifact caches after the repo (e.g.
  test_report_dask__distributed) so that reports for multiple repos can be
  generated from the same working directory.
- Fall back to `gh auth token` when GITHUB_TOKEN is not set.
- On PRs, run the workflow only when the `test-report` label is set; upload
  the reports and databases as workflow artifacts instead of deploying to
  GitHub Pages.
- Fix actions/cache usage so that the database cache is actually updated on
  every run (cache keys are immutable).
- Skip in-progress and expired runs/artifacts to avoid poisoning the cache.

Co-Authored-By: Claude Fable 5 <[email protected]>
@crusaderky crusaderky added the test-report Run the test-report workflow on this PR label Jul 3, 2026
The explicit permissions block removes the default actions:read scope, which
is needed to list workflow runs and download artifacts. The PR run passed
regardless because fork PR tokens are read-only on all scopes.

Co-Authored-By: Claude Fable 5 <[email protected]>
@github-actions

github-actions Bot commented Jul 3, 2026 •

Copy link
Copy Markdown
Contributor

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    40 files  ±0      40 suites  ±0   14h 41m 23s ⏱️ - 16m 20s
 4 159 tests ±0   3 981 ✅ ±0    178 💤 ±0  0 ❌ ±0 
80 941 runs  ±0  76 704 ✅ +1  4 237 💤  - 1  0 ❌ ±0 

Results for commit 69f7224. ± Comparison against base commit 0f72f8f.

♻️ This comment has been updated with latest results.

crusaderky and others added 4 commits July 3, 2026 16:23
Each job gets strictly the permissions it needs: generate gets actions:read
(plus contents:read for checkout), deploy gets contents:write. The report is
handed over between the jobs as a workflow artifact, which also becomes
always available for debugging; the databases are only uploaded on PRs.

Co-Authored-By: Claude Fable 5 <[email protected]>
Public repositories are implicitly readable by the workflow token even when
the permissions block sets contents to none.

Co-Authored-By: Claude Fable 5 <[email protected]>
e.g. test_report.dask__distributed.db. This simplifies .gitignore and the
workflow glob patterns, which previously had to avoid matching
test_report.html.

Co-Authored-By: Claude Fable 5 <[email protected]>
@crusaderky crusaderky changed the title PLEASE IGNORE: unsupervised AI pr — Repair test-report tooling [unsupervised AI] Repair test-report tooling Jul 9, 2026
@crusaderky crusaderky changed the title [unsupervised AI] Repair test-report tooling Repair test-report tooling Jul 9, 2026
@crusaderky
crusaderky marked this pull request as ready for review July 9, 2026 17:03
@crusaderky
crusaderky requested a review from fjetter as a code owner July 9, 2026 17:03
@crusaderky
crusaderky merged commit 462a51a into dask:main Jul 9, 2026
52 of 53 checks passed
@crusaderky
crusaderky deleted the test-report branch July 9, 2026 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test-report Run the test-report workflow on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant