Skip to content

Don't leave Nanny stuck in closing status when worker kill times out (fix flaky test_chaos_rechunk) - #9317

Merged
crusaderky merged 2 commits into
dask:mainfrom
crusaderky:test_chaos_rechunk
Jul 9, 2026
Merged

Don't leave Nanny stuck in closing status when worker kill times out (fix flaky test_chaos_rechunk)#9317
crusaderky merged 2 commits into
dask:mainfrom
crusaderky:test_chaos_rechunk

Conversation

@crusaderky

@crusaderky crusaderky commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

test_chaos_rechunk frequently hangs, e.g. https://github.com/dask/distributed/actions/runs/28594679995/job/84786875121?pr=9315

This is due to an underlying race condition in Nanny which is triggered on very slow hosts (like Windows CI frequently is).

@github-actions

github-actions Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

   19 files   -     21     19 suites   - 21   7h 8m 50s ⏱️ - 7h 6m 35s
  100 tests  -  4 055    100 ✅  -  3 870    0 💤  -   180  0 ❌  - 5 
1 900 runs   - 77 520  1 800 ✅  - 73 412  100 💤  - 4 103  0 ❌  - 5 

Results for commit b1bef38. ± Comparison against base commit 2d43f7f.

This pull request removes 4155 and adds 100 tests. Note that renamed tests count towards both. 
distributed.cli.tests.test_dask_scheduler ‑ test_dashboard
distributed.cli.tests.test_dask_scheduler ‑ test_dashboard_allowlist
distributed.cli.tests.test_dask_scheduler ‑ test_dashboard_non_standard_ports
distributed.cli.tests.test_dask_scheduler ‑ test_dashboard_port_zero
distributed.cli.tests.test_dask_scheduler ‑ test_defaults
distributed.cli.tests.test_dask_scheduler ‑ test_hostport
distributed.cli.tests.test_dask_scheduler ‑ test_idle_timeout
distributed.cli.tests.test_dask_scheduler ‑ test_interface
distributed.cli.tests.test_dask_scheduler ‑ test_multiple_protocols
distributed.cli.tests.test_dask_scheduler ‑ test_multiple_workers
…

distributed.tests.test_stress ‑ test_chaos_rechunk[1-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[10-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[100-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[11-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[12-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[13-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[14-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[15-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[16-100]
distributed.tests.test_stress ‑ test_chaos_rechunk[17-100]
…

♻️ This comment has been updated with latest results.

@crusaderky crusaderky mentioned this pull request Jul 3, 2026
Closed
4 tasks
@crusaderky crusaderky added flaky test Intermittent failures on CI. tests Unit tests and/or continuous integration labels Jul 9, 2026
@crusaderky crusaderky changed the title Fix flaky test_chaos_rechunk [Unsupervised AI] Fix flaky test_chaos_rechunk Jul 9, 2026
@crusaderky crusaderky changed the title [Unsupervised AI] Fix flaky test_chaos_rechunk [unsupervised AI] Fix flaky test_chaos_rechunk Jul 9, 2026
@crusaderky
crusaderky force-pushed the test_chaos_rechunk branch from b1bef38 to 5b013ee Compare July 9, 2026 13:52
@crusaderky crusaderky changed the title [unsupervised AI] Fix flaky test_chaos_rechunk Fix flaky test_chaos_rechunk Jul 9, 2026
@crusaderky
Don't leave Nanny stuck in closing status when worker kill times out
5c735a7 
Fix test_chaos_rechunk flakiness on slow CI hosts with a Nanny stuck in Status.closing at teardown.
@crusaderky
crusaderky force-pushed the test_chaos_rechunk branch from 5b013ee to 5c735a7 Compare July 9, 2026 13:58
@crusaderky crusaderky changed the title Fix flaky test_chaos_rechunk Don't leave Nanny stuck in closing status when worker kill times out (fix flaky test_chaos_rechunk) Jul 9, 2026
@crusaderky
crusaderky marked this pull request as ready for review July 9, 2026 13:58
@crusaderky
crusaderky requested a review from fjetter as a code owner July 9, 2026 13:58
@crusaderky
crusaderky merged commit 90e1a84 into dask:main Jul 9, 2026
44 of 48 checks passed
@crusaderky
crusaderky deleted the test_chaos_rechunk branch July 9, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fjetter fjetter Awaiting requested review from fjetter

Assignees

No one assigned

Labels

flaky test Intermittent failures on CI. tests Unit tests and/or continuous integration

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@crusaderky