Skip to content

High latency can cause get_task_stream() to miss tasks #9253

Description

@amotl

Describe the issue:

When submitting a task after starting a cluster, it is not available per get_task_stream() inquiry. If we "ping" the cluster first by invoking an initial get_task_stream(), everything is accounted for perfectly afterwards.

Minimal Complete Verifiable Example:

#!/usr/bin/env python
from dask.distributed import Client


def main():
    with Client() as client:

        # FIXME: The program succeeds only when this line is active.
        # tasks = client.get_task_stream()

        # Submit dummy task.
        client.submit(lambda: True).result()

        # Inquire tasks and validate the submitted task has been run.
        tasks = client.get_task_stream()
        if not tasks:
            raise RuntimeError("No tasks have been recorded")
        task = tasks[-1]
        assert task["key"].startswith("lambda"), f"Task has wrong key: {task['key']}"
        assert task["status"] == "OK", f"Task status not OK: {task['status']}"
        print("SUCCESS")


if __name__ == "__main__":
    main()

Anything else we need to know?:

We have been running into this on behalf of those downstream exercises:

Environment:

  • Dask version: 2026.3.0
  • Python version: 3.13
  • Operating System: macOS / Linux
  • Install method (conda, pip, source): OCI / conda / uv

Kudos:

Thank you for conceiving Dask. It's a true gem.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is brokendiagnosticsp3Affects a small number of users or is largely cosmetic

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions