Repository navigation
matplotlib dev pulls in contourpy that pins numpy<2, preventing numpy dev testing #26847
Closed as not planned
Description
Hello. Looks like contourpy started pinning numpy<2 in both their dev branch and releases. matplotlib pulls in contourpy as a required dependency. As a result, that forces CI to downgrade numpy to 1.26 even when I am asking for numpy "nightly" wheel in an environment where I also asks for matplotlib nightly wheel.
Is it possible for matplotlib to coordinate with contourpy developers so matplotlib dev does not end up forcing numpy to downgrade from 2.0.dev to 1.x? 🙏
Latest release:
Dev:
Activity
pllim commentedon Sep 20, 2023
p.s. I bet you have this same problem in your CI unless you force numpy to upgrade again afterwards but your CI seems to skip nightly installs for pushes to
main.QuLogic commentedon Sep 20, 2023
cc @ianthomas23
ksunden commentedon Sep 20, 2023
Pinning is the official numpy recommendation, at least for releases (at least for users of the C API, because the problem is that they are anticipating an ABI break)
We have elected here to pin on release branches only, and leave the
mainbranch unpinned.I suspect some of it can be worked around with careful ordering and/or
pip install --no-build-isolation --no-depstrickery, though mildly frustrating.I expect when the ABI breaks happen we will have to sort out nightlies regardless though, as our wheels uploaded there are made with current released numpy, not nightly numpy...
(And even then may require a recompiled contourpy as well as mpl)
pllim commentedon Sep 20, 2023
So, are you suggesting that I cannot install
matplotlibif I were to test against numpy2.0.dev? 🤔p.s. For reasons beyond my control,
astropyusestoxto manage test environments. I don't know an easy way to force numpy to upgrade again after the downgrade using their mini-language. Also, since you are not testing against numpy dev, then there is also a risk of astropy picking up incompatibility between matplotlib and numpy instead of astropy stuff.ksunden commentedon Sep 20, 2023
At present it all works.
Once the ABI break hits, the wheels that we currently upload to nightlies are not built against np2.0. Since the ABI break has not happened yet, this is just something that hasn't been an issue.
So to test against numpy 2.0 (post ABI break) you will need mpl compiled against 2.0, which either means doing a source install (rather than a wheel install) or our nightlies will have to be updated to compile against np2.0.
Currently we just upload the most recent run from main of our
CIBWaction, which uses current released numpy. (Until recently it actually usedoldest_supported_numpy, but since np 1.25, backwards compatible mode is available and default, so now there is no reason to build against older versions)but if/when numpy nightly is ABI incompatible with released numpy (which is expected, and the reason for pinning), then we may have to rethink what we do for nightly wheel uploads.
QuLogic commentedon Sep 20, 2023
Nightly tests are on a schedule. And we do test against NumPy 2 because nightly deps are installed after our dependencies, and Matplotlib itself is installed with
--no-deps. We don't build against it though, but that's because of build isolation, which will be turned off with #26621.pllim commentedon Sep 21, 2023
What are your options?
In the meantime, I can probably separate out testing of matplotlib dev and numpy dev into different jobs, but I can see the tech debt hitting us downstream in the long run.
ksunden commentedon Sep 21, 2023
I think one option would be to build our nightlies using the numpy nightlies rather than using numpy stable.
That would probably mean more complicated CI on our end since I think we would have to have a job that specifically builds that to upload as I don't think we want the job that runs on main/our release process to use nightlies for building our wheel artifacts.
It's not 100% clear to me what the ABI break will look like. They have said something about it being "behind a feature flag", so it may be that the nightlies continue to be ABI compatible for most of the cycle and its only a small amount at the end that we have to worry about it at all, or it may be that nightlies are where they flip the switch early and we need to adapt sooner.
ianthomas23 commentedon Sep 21, 2023
This made me laugh! In my experience most Open Source involvement is at least mildly frustrating.
The
numpy<2pin incontourpyis following the official recommendation.To test
contourpyagainstnumpy2.0 in CI there is a post-build step ofwhich works fine to test
contourpyagainstnumpy>=2.I figure other projects need to do something similar. If anyone has any better suggestions I am all ears.
pllim commentedon Sep 21, 2023
Brute force reinstall does work, but it resulted in warning from pip resolver, and duplicate code. Not pretty but better than nothing. spacetelescope/jdaviz#2472
story645 commentedon Sep 21, 2023
Total tangent but really liking the trivial label and thinking we should adopt something similar/make more use of our only needs one review label.
tacaswell commentedon Oct 3, 2023
I'm going to close this as something we can not fix at the Matplotlib level (because it is pip respecting the pinning of released versions of down stream packages who are following a shared upstream's pinning request).
For better or worse, if you want to live on the edge of things sometimes you have to fight the packaging system a bit.
neutrinoceros commentedon Feb 28, 2024
contourpy is now publishing nightly builds without the pin
xref contourpy/contourpy#362