Repository navigation
[Python] Compatibility with NumPy 2.0 #39532
Description
NumPy 2.0 is coming soon, and we should ensure pyarrow is fully compatible to ensure a smooth transition of our users. Overview issue on numpy's side: numpy/numpy#24300
We already test with nightly numpy (eg https://github.com/ursacomputing/crossbow/actions/runs/7454906275/job/20283098411), and currently this build is working fine.
There is one warning about the usage of numpy.core.numerictypes that we should still solve.
Further, we need to check if we are all OK on the packaging side: do we need to pin numpy<2 for now in our upcoming release? What do we need to update in our wheel building? (stop using oldest-supported-numpy?)
- [Python] NumPy 2.0 compat: remove usage of np.core #39533
- [Packaging][Python] Add a numpy<2 pin to the install requirements in the release branch #39537
- [Packaging][Python] Enable building pyarrow against numpy 2.0 #39555
- Ensure we can build pyarrow against numpy 2.0 nightly (update pyproject.toml to allow this), and test this by building our nightly wheels with numpy nightly
- [CI][Python] The pandas upstream_devel nightly build is skipping all pandas tests #39536
- [Python][Packaging] Remove nightly numpy wheels from the wheel build requirements #39848
- [Python] Adapt to changes in the
copykeyword for numpy 2.0+ compatibility #41098 - [Python] NumPy 2 ABI changes (elsize) #40376
cc @raulcd
Activity
jorisvandenbossche commentedon Jan 9, 2024
Some other observations / relevant bits from the linked issues:
Since numpy 1.25, we don't need
oldest-supported-numpyanymore (https://numpy.org/devdocs/release/1.25.0-notes.html#compiling-against-the-numpy-c-api-is-now-backwards-compatible-by-default and https://github.com/scipy/oldest-supported-numpy#deprecation-notice-for-numpy-20), so for newer Python versions (where the minimum supported version of numpy is <=1.25), we can start compiling with the latest numpyAt the moment, pip install pyarrow in an environment with numpy 2.0, the import of pyarrow fails. So at a minimum, we should add a
numpy<2.0pin in the release branchIf we want wheels that support both numpy 1.x and 2.0 at runtime, we need to build our wheels with numpy 2.0 (https://numpy.org/devdocs/dev/depending_on_numpy.html#numpy-2-0-specific-advice). The recommendation is to start doing that (and then a numpy<2.0 pin in the released version is no longer needed) once numpy 2.0rc1 is out. This is targeted for around 1 Feb 2024, but other projects already started building their wheels with numpy 2.0.dev, as far as I understand (eg BLD: Build scikit-learn nightlies using NumPy nightlies scikit-learn/scikit-learn#27735, or that is only for the nightly wheels?)
Updating the top-post task list with some TODOs
GH-39555: [Packaging][Python] Enable building pyarrow against numpy 2…
apacheGH-39555: [Packaging][Python] Enable building pyarrow against n…
apacheGH-39532: [Python] Add copy keyword in Array.__array__ for nump…
Merge remote-tracking branch 'upstream/main' into apachegh-39532-nump…
Merge remote-tracking branch 'upstream/main' into apachegh-39532-nump…
GH-41098: [Python] Add copy keyword in Array.__array__ for numpy 2.0+…
GH-41098: [Python] Add copy keyword in Array.__array__ for numpy 2.0+…
raulcd commentedon Apr 15, 2024
@jorisvandenbossche is there something else to do on this one or are we ready for the release on this?
jorisvandenbossche commentedon Apr 15, 2024
I think all sub-tasks are done, and this can be closed now
apacheGH-41098: [Python] Add copy keyword in Array.__array__ for nump…