WHL: build and distribute stable-ABI-compliant wheels - #18420

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

• Do the proposed changes actually accomplish desired goals?
• Do the proposed changes follow the Astropy coding guidelines?
• Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
• Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
• Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
• Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
• Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
• Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
• At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

Do we still need 313t spec if there is 314 now?

Yes. The current stable abi doesn't work on 3.13t or 3.14t (which is one of the reasons why we wanted to make it optional), and we can't yet move this job to 3.14t because none of our deps have wheels on PyPI yet.
but while we're discussing this, it's worth mentionning that CPython devs want the next stable abi to be free-threading compatible, so, assuming we can comply with it, we could get 3.15t-instalable wheels for free when we add support for 3.15

I don't think change log is needed but release manager can always overrule.

Thanks!

Surprisingly, this works as I expect on macOS and windows (all archs), but not on Linux: all jobs produced 3 wheels each (one for each Python version), never tagging the ABI as abi3. Before investigating further, let me ask @astrofrog if he thinks this could be a simple bug in extensions-helper ?

never mind I just remembered I saw something similar once; I think it has to do with os-specific ways envs are isolated, and can easily be worked-around in cibuildwheel (once you know what to look for). Let me give this another shot.

Now the env var is correctly passed to the build env on all platforms, however, building currently fails on Linux. It looks to me like the env var isn't having all desired effects, in particular, it seems like the Py_LIMITED_API macro is somehow defined too late (i.e., after Python.h was already included). I don't yet understand why this problem wasn't caught in existing CI and what's the important difference in a cibuildwheel context.
However, it might be obvious to @henryiii, so let me try my luck :)

Actually I may have been misreading the logs, I think they were showing genuine compile-time issues (that do not necessarily show up as fatal errors on all compilers, hence why we didn't see them before ?). Any way, I'll try something new and take the compilers seriously for now.

I'm getting there. Letting CI run 100% one last (🤞🏻) time.

Alright, this is now ready for review.

Don't think there's any reason not to merge, might as well start exercise this for the dev wheels. We can easily disable the limited API wheels again if needed.

Thank you for your hard work on this!

Yes thank you for the epic effort on this!

Just to close the loop on this - it all works fine! See dev wheels here:

https://anaconda.org/astropy/astropy/files?type=pypi&version=7.2.dev288%2Bgd02ffa90d6

The wheel building now takes 1h45 instead of 2h15 roughly, so saves 30 minutes on every wheel build job. The reason it hasn't shaved off more is because we still run the test suite on all versions, but I think that's probably sensible for now.

That's actually the recommendation from Python's docs:

we recommend testing an extension with all minor Python versions it supports, and preferably to build with the lowest such version.