Repository navigation
Automate build and release to PyPI #7390
Description
Activity
This sounds like a great step.
Love it!!
- pinned this issue
on Nov 12, 2023 I've opened #7544 to create the sdist during the release build.
Next, I'd like us to move the Windows wheel building from
test-windows.ymlinto the samewheels.ymlworkflow.So we get something like this:
And the Windows wheels should be part of the same
dist.zipartifact containing the Linux and macOS wheels, and the sdist.I think we also probably want to use a reusable workflow, similar to
wheels-linux.ymlandwheels-macos.yml.We don't need to build the wheels anymore in
test-windows.yml. If there's a lot of common steps needed for testing (intest-windows.yml) and building wheels, a reusable workflow could be useful here.This doesn't all need to be done in one go, incremental steps are fine, as long as they're atomic changes and we're still able to make releases during any intermediate step.
Any thoughts on how to approach this? cc @nulano
After all this is done, we have all* the release files created in one place on the CI, and do the upload to PyPI! But one step at a time :)
(* All the GHA files, not Travis CI, which will still need manual upload, but they should be ready before all the automation.)
Sounds good! When the time comes, we can ask around for someone to test the ARM64 wheels :)
I've since realized that I can borrow a relative's M2 MacBook to test the wheels.
The Python 3.12 wheel from my WIP branch tested OK.
I then usedcibuildwheelmanually in the terminal to build and test wheels for the 10.1.0 tag for #7547 (comment)Reacted by Hugo van Kemenade and Jeffrey 'Alex' Clark#7616 has now been merged.
@aclark4life could you go to https://pypi.org/manage/project/pillow/settings/publishing/ and add a publisher?
Owner: python-pillow
Repository name: Pillow
Workflow name: wheels.yml
Environment name: release-pypiReacted by Jeffrey 'Alex' Clark@aclark4life Please can you also set up an environment named
release-pypiin this repo, that all the release managers can access: at least @radarhere and me, probably also you and @wiredfool for backup.Reacted by Jeffrey 'Alex' Clark@radarhere @hugovk Thanks and done, I think! The env
release-pypialready exists … need me to do anything to it?
Thanks, the publisher looks good!
The env
release-pypialready exists … need me to do anything to it?@aclark4life What settings does it have?
@radarhere What settings should it have?
Most notably, no environment vars or secrets configured.
The publisher looks good to me, thanks. The details I mentioned were the only ones I added.
So when releasing, after pushing the tag, the CI will begin, build the sdist and all the wheels, and somehow put them in an environment? How does that look? What's the next step to upload to PyPI?
Probably need a testpypi release first.
When I was using Test PyPI in #7616, once sdist and the wheels were built, the action uploaded them, and that was it.
https://github.com/radarhere/Pillow/actions/runs/7329629479/job/19960359962#step:4:1008
I presume that Test PyPI works the same way as real PyPI. I would have imagined that real PyPI releases don't rely on test PyPI to go first, since we've been ignoring test PyPI for every previous release we've done.
Okay, so the environment isn't doing anything. Anyway, I'm excited for the next release!
@radarhere Please will you update https://github.com/python-pillow/Pillow/blob/main/RELEASING.md? It can wait until after doing the release if you like.
The tagged "Upload release to PyPI" ran smoothly in the Pillow 10.2.0 release.
With that, is this completed?
Reacted by Hugo van KemenadeGreat to hear it went smoothly!
Before closing this issue, we still have Travis to automate. Shall we try my suggestion at #7485 (comment) to upload to PyPI directly from Travis?
Sure, sounds good if it works.
Only loosely following, but I think I like Plan C!! Thanks all ❤️ (Losing travis in exchange for longer build times is my "hot take", sounds worth it to me.)
Reacted by Hugo van Kemenade- unpinned this issue
on Jan 4, 2024
Currently the release process has many steps, including preparing things in this repo, and then starting a build in https://github.com/python-pillow/pillow-wheels, before downloading the sdist + wheel artifacts to disk, and manually uploading them to PyPI using twine.
I'd like us to move towards more automated releases, including using the new Trusted Publishers to upload wheels directly from the CI to PyPI, which is more secure:
We can do this in a phased manner, not all of these things would need to be done at once, to ensure we can still make releases with just some of them completed.
We could potentially deal with the Travis CI wheels later, and tackle GitHub Actions first as they're a bigger win and we don't know when Travis CI will go away.
Big picture:
I'd suggest something along these lines:
Move Linux and macOS https://github.com/python-pillow/pillow-wheels to this repo https://github.com/python-pillow/Pillow. We already have Windows wheels build here.
paths:/paths-ignoreto workflows so we only trigger the main test workflows for "normal" code changes, and only trigger the wheel workflow for things like release and modifying its configAdd automation to download artifacts and upload them to PyPI via Trusted Publishing
Thoughts?