Add PyPI release workflow by mrocklin · Pull Request #12452 · dask/dask

mrocklin · 2026-06-09T14:15:15Z

This moves the release procedure from a manual process done on a laptop to automated on GitHub Actions. This uses the Trusted Publisher functionality on PyPI that has, I think, become somewhat standard. There's still a manual approval process in the chain.

When reviewing I recommend looking at the release procedure doc.

For an example run see https://github.com/mrocklin/dask/actions/runs/27209613416

See dask/community#444

Sibling PR for distributed at dask/distributed#9297

crusaderky

Thanks for the PR! It mostly looks good. A few notes below.

crusaderky · 2026-06-09T14:31:22Z

+          release_version="$RELEASE_VERSION"
+          if [[ ! "$release_version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+              echo "::error::Release versions must use x.x.x form, got $release_version"


Suggested change

release_version="$RELEASE_VERSION"

if [[ ! "$release_version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then

echo "::error::Release versions must use x.x.x form, got $release_version"

if [[ ! "$RELEASE_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then

echo "::error::Release versions must use x.x.x form, got $RELEASE_VERSION"

crusaderky · 2026-06-09T14:36:26Z

+      - name: Check built versions
+        run: |
+          python - <<'PY'
+          import email
+          import glob
+          import os
+          import tarfile
+          import zipfile
+
+          release_version = os.environ["RELEASE_VERSION"]
+          major, minor, patch = release_version.split(".")
+          next_patch_version = f"{major}.{minor}.{int(patch) + 1}"
+
+          wheel_paths = glob.glob("dist/*.whl")
+          sdist_paths = glob.glob("dist/*.tar.gz")
+          if len(wheel_paths) != 1 or len(sdist_paths) != 1:
+              raise SystemExit("Expected exactly one wheel and one sdist")
+
+          with zipfile.ZipFile(wheel_paths[0]) as wheel:
+              metadata_path = next(name for name in wheel.namelist() if name.endswith(".dist-info/METADATA"))
+              wheel_metadata = email.message_from_bytes(wheel.read(metadata_path))
+              wheel_version = wheel_metadata["Version"]
+
+          with tarfile.open(sdist_paths[0], "r:gz") as sdist:
+              pkg_info_path = next(name for name in sdist.getnames() if name.endswith("/PKG-INFO"))
+              pkg_info = sdist.extractfile(pkg_info_path)
+              if pkg_info is None:
+                  raise SystemExit("Could not read sdist PKG-INFO")
+              sdist_metadata = email.message_from_binary_file(pkg_info)
+              sdist_version = sdist_metadata["Version"]
+
+          for artifact, version in {"wheel": wheel_version, "sdist": sdist_version}.items():
+              if version != release_version:
+                  raise SystemExit(f"{artifact} version {version} does not match release {release_version}")
+
+          for artifact, metadata in {"wheel": wheel_metadata, "sdist": sdist_metadata}.items():
+              requirements = metadata.get_all("Requires-Dist") or []
+              distributed_requirements = [
+                  requirement
+                  for requirement in requirements
+                  if requirement.startswith("distributed") and 'extra == "distributed"' in requirement
+              ]
+              if len(distributed_requirements) != 1:
+                  raise SystemExit(
+                      f"{artifact} should declare exactly one distributed extra requirement, "
+                      f"got {distributed_requirements}"
+                  )
+              requirement = distributed_requirements[0].replace(" ", "")
+              if f">={release_version}" not in requirement or f"<{next_patch_version}" not in requirement:
+                  raise SystemExit(
+                      f"{artifact} distributed extra should require distributed>={release_version},"
+                      f"<{next_patch_version}, got {distributed_requirements[0]}"
+                  )
+          PY


IMHO this step is quite overcomplicated and redundant with smoke_test_pypi.
I would suggest replacing it with a simpler bash step at the end of the smoke_test_pypi workflow that tests that both of these commands return the expected version:

python -c import dask; print(dask.__version__)

pip freeze | sed -n 's/^dask==//p'

Fair point. Thanks.

crusaderky · 2026-06-09T14:39:47Z

+  dry_run_publish_pypi:
+    name: Dry-run PyPI publish
+    needs: smoke_test_pypi
+    if: github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/download-artifact@v7
+        with:
+          name: dist
+          path: dist
+      - name: Confirm publish skipped
+        run: |
+          ls -l dist
+          echo "Dry run only; not publishing dask==$RELEASE_VERSION to PyPI."
+


This whole block for the sake of an echo feels unnecessary?

Suggested change

dry_run_publish_pypi:

name: Dry-run PyPI publish

needs: smoke_test_pypi

if: github.event_name == 'workflow_dispatch'

runs-on: ubuntu-latest

steps:

- uses: actions/download-artifact@v7

with:

name: dist

path: dist

- name: Confirm publish skipped

run: |

ls -l dist

echo "Dry run only; not publishing dask==$RELEASE_VERSION to PyPI."

crusaderky · 2026-06-09T14:46:34Z

-    *   dask version in distributed/pyproject.toml
-    *   pins should be >= the current version but < the next version to allow 
+*   Update dependency bounds in all pyproject.toml files
+    *   `distributed` dependency in pyproject.toml of dask/dask


We should clarify that in dask/pyproject.toml, you need to set e.g.

[project.optional-dependencies] distributed = ["distributed >=2026.6.0,<2026.6.1"] ...

while in distributed/pyproject.toml you'll need to have e.g.

dependencies = [ "dask >=2026.6.0,<2026.6.1", ... ]

Which means that you can do the dask release with the impossible distributed pin, thanks to the fact that distributed is an optional dependency, but vice versa, you can't release distributed 2026.6.0 before dask 2026.6.0 has been published, because for distributed dask is a hard dependency.

Done. In the future I wouldn't mind finding a way to automate this as well. It feels error prone. Future work though.

crusaderky · 2026-06-09T14:48:30Z

+    *   In `dask/distributed`, run the `Release Publisher` workflow manually
+        with the Distributed version to rehearse and a Dask version that is
+        already available on PyPI.


import distributed against the previous dask version is at risk of failing to import.
I can't see an easy way out. I think it should be clarified here that it's not the end of the world if this fails.

What matters a lot more is the most recent Upstream/py314 test run in dask/distributed (it can also be triggered by hand), which automatically detects breakages in dask/distributed unit tests caused by recent changes in dask/dask.

crusaderky · 2026-06-09T14:54:19Z

+    versions, publishes them to PyPI with Trusted Publishing, and publishes the
+    GitHub Release.
+
+    GitHub Actions pauses at the `pypi` environment for manual approval. Open


I can't find this pause in the publish_pypi job. Is it an implicit default in pypa/gh-action-pypi-publish@release? If so I'd rather make it explicit.

This will be set in GitHub configuration under the pypi Environment settings. I do this in other projects and it works well.

crusaderky · 2026-06-09T14:56:34Z

+    PyPI publishing skips files that already exist, so rerunning the workflow
+    can recover after a partial success such as a GitHub Release failure.


dask and distributed feature a single artifact each though; what's the benefit of this?

The Github release is downstream of the pypi release in the workflow. There's a world where we succeed in publishing a PyPI release but fail somehow to publish a github release. We'd need to redo the pipeline for this and pass over the PyPI stage. Very open to other ideas here. I'll admit to not caring too much about the GitHub release.

mrocklin · 2026-06-09T15:48:07Z

Thanks for the review @crusaderky . Pushed changes. Re-review welcome.

github-actions · 2026-06-09T16:54:16Z

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

25 files ±0 25 suites ±0 7h 4m 25s ⏱️ - 4m 54s
18 639 tests ±0 17 450 ✅ ±0 1 189 💤 ±0 0 ❌ ±0
379 001 runs ±0 331 757 ✅ +1 47 244 💤 - 1 0 ❌ ±0

Results for commit ba7ee8e. ± Comparison against base commit d0666ee.

mrocklin · 2026-06-10T23:19:46Z

Planning to merge this in tomorrow and do the release. Please speak up if anyone disagrees.

crusaderky

I have not tested it, but otherwise it looks good. Thank you!

Add PyPI release workflow

40f74ab

This was referenced Jun 9, 2026

Add PyPI release workflow dask/distributed#9297

Merged

Release 2026.6.0 dask/community#444

Closed

crusaderky reviewed Jun 9, 2026

View reviewed changes

Simplify PyPI release workflow

ba7ee8e

crusaderky approved these changes Jun 11, 2026

View reviewed changes

mrocklin merged commit aea7965 into dask:main Jun 11, 2026
33 of 34 checks passed

mrocklin deleted the release-procedure branch June 11, 2026 17:05

		PyPI publishing skips files that already exist, so rerunning the workflow
		can recover after a partial success such as a GitHub Release failure.

Conversation

mrocklin commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

crusaderky left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crusaderky Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crusaderky Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mrocklin commented Jun 9, 2026

Uh oh!

github-actions Bot commented Jun 9, 2026

Unit Test Results

Uh oh!

mrocklin commented Jun 10, 2026

Uh oh!

crusaderky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mrocklin commented Jun 9, 2026 •

edited

Loading

crusaderky Jun 9, 2026 •

edited

Loading

crusaderky Jun 9, 2026 •

edited

Loading