Reduce size of scheduler_info - #9045

This changed in dask/distributed#9045. We always
want everything.

For no good reason other than making Jupyter pretty printing nicer,
dask/distributed#9045 broke the API of
`client.scheduler_info()` by changing the default number of workers that
are retrieved to `5`.

We use this information in various places to ensure the correct number
of workers have connected to the scheduler, and thus this change now
specifies `n_workers=-1` to get all workers instead of just the first
`5`. This happened not to be seen in CI previously because we don't run
multi-GPU tests, let alone with more than `5` workers. Running various
tests from `test_dask_cuda_worker.py` on a system with 8 GPUs will cause
them to timeout while waiting for the workers to connect.

For no good reason other than making Jupyter pretty printing nicer, dask/distributed#9045 broke the API of `client.scheduler_info()` by changing the default number of workers that are retrieved to `5`.

We use this information in various places to ensure the correct number of workers have connected to the scheduler, and thus this change now specifies `n_workers=-1` to get all workers instead of just the first `5`. This happened not to be seen in CI previously because we don't run multi-GPU tests, let alone with more than `5` workers. Running various tests from `test_dask_cuda_worker.py` on a system with 8 GPUs will cause them to timeout while waiting for the workers to connect.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Jacob Tomlinson (https://github.com/jacobtomlinson)

URL: #1514

This fixes hangs when there are more than 5 workers in the dask cluster
(as a consequence of dask/distributed#9045;
previously that returned all the worker addresses in 'workers', now it's
limited to the top 5 by default).

This fixes hangs when there are more than 5 workers in the dask cluster (as a consequence of dask/distributed#9045; previously that returned all the worker addresses in 'workers', now it's limited to the top 5 by default).

Authors:
  - Tom Augspurger (https://github.com/TomAugspurger)

Approvers:
  - Rick Ratzel (https://github.com/rlratzel)

URL: #5147

Following review feedback: `scheduler_info()` previously returned up to 5
workers for asynchronous clients, since they read the periodically refreshed
`_scheduler_identity` cache rather than fetching on demand (a sync method
can't await inside the event loop). Returning an arbitrary five workers is
misleading.

The cached identity exists only to feed the client repr, which needs the
cluster-wide totals (`n_workers`, `total_threads`, `total_memory`) — and those
are maintained as scheduler-level aggregates, independent of the worker dict.
So the periodic `_update_scheduler_info` now fetches `n_workers=0`: totals are
still accurate, and `scheduler_info()["workers"]` is empty for async clients
rather than a misleading subset. Synchronous clients are unaffected (they
refetch with all workers by default). This also shrinks the periodic payload
further, reinforcing the scheduler-scalability fix from dask#9045.

Callers that need per-worker detail on an async client fetch it explicitly via
`await client.scheduler.identity(n_workers=-1)`; async tests that relied on the
cached workers are updated accordingly.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

`Client.scheduler_info()` defaulted to `n_workers=5`, silently truncating
the "workers" dict to the first five workers. This was misleading and, more
importantly, broke `restart_workers`: both the nanny-required validation and
the worker name -> address resolution read this truncated set, so on clusters
with more than five workers the nanny check was silently skipped and
name-based restarts of later workers misfired.

Changes:
- `scheduler_info()` now defaults to `n_workers=-1` (all workers). This only
  affects explicit, on-demand calls, not the periodic per-client poll, so the
  scheduler-scalability fix from #9045 / #9043 is not reintroduced (the
  periodic cache populator stays capped at 5).
- `_restart_workers` now fetches a fresh, full identity via
  `scheduler.identity(n_workers=-1)` and performs both the nanny check and the
  name resolution there, correct for any cluster size. This also collapses the
  previous two identity RPCs on the sync path into one.
- Document the sync/async asymmetry: async clients read the periodically
  cached value, so `n_workers` is only honored for synchronous clients.

Adds a regression test that restarts a non-nanny worker beyond the historical
five-worker cap; it raises the expected error only with the fix.

xref #9065

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Co-authored-by: Guido Imperiale <[email protected]>