Tracking issue: defaulting to Meson - follow-up tasks

After gh-23838 we now default to building with Meson (via meson-python, when invoking a build with pip install . or similar) in main. There are a number of more niche build configs not yet ported, and a few things were xfail-ed and need fixing up. Here is a list, roughly in order of priority:

Build nightlies for Python 3.12 to unblock other projects from testing with 3.12 (xref CI: BLD: build wheels and fix test suite for Python 3.12 #23991)
Improve how CBLAS and ILP64 flags/dependencies are handled (xref BLD: undefined symbol: cblas_sgemm when building with meson with pip install #23909, and also automate passing CFLAGS/CXXFLAGS - I'll work on this next): see BLD: improve handling of CBLAS, add -Duse-ilp64 build option #23984
Finish implementing all BLAS/LAPACK auto-selection and control options (e.g. the equivalent of NPY_BLAS_ORDER). Done in ENH: meson: implement BLAS/LAPACK auto-detection and many CI jobs #24893
Fix up the issue with PyPy and re-enable the pypy3.9 CI job on Azure. Done in BLD/CI: re-enable ILP64 usage and PyPy job in Azure #24238.
Integrate SIMD support (the relevant CI jobs are still going through runtests.py)
Update the build docs (partially done in DOC: Update building docs to use Meson #24573; the rest in DOC: add "building from source" docs #25350).
Fix and re-eanble the 32-bit Windows wheels builds with MSVC on Azure (or move to GHA, or drop? - see DEP: stop building and testing 32-bit windows #23717). Re-introducing on GHA in CI/BLD: fail by default if no BLAS/LAPACK, add 32-bit Python on Windows CI job #24279.
Fix the long double representation check so the builds works on some of the more niche platforms again (xref BUG: New meson system fails to identify long double representation on 64-bit BE platforms #23972)
Fix the xfail-ed test_cython for Windows due to a missing npyrandom.lib (done in BLD: change file extension for installed static libraries back to .lib #23983)
Move Travis CI jobs over to Meson
Check that all CI jobs have been moved over (e.g., by removing runtests.py and setup.py, then running CI and ensuring nothing fails). Done - there were a lot left. See Restructuring GitHub Actions CI jobs #24410 for the tracking issue for CI migration.
Investigate how to fix the bin/f2py3X CLI install, in addition to the f2py and f2py3 ones (see project.scriptsinpyproject.toml`, and commit f22a33b). Or can/should that interface be dropped? Multiple installs into a single env is considered bad practice by now. Dropped in BLD: only install the f2py command, not f2py3 or f2py3.X #24235.
Investigate and fix the two einsum test failures with int8/uint8 that were xfailed in BLD: switch to meson-python as the default build backend #23838 (EDIT: these two tests fail with clang-cl on Windows too, see ENH: Add changes that allow NumPy to compile with clang-cl #20866 (comment)). EDIT: opened separate issue: BUG: einsum with int8/uint8 input is buggy #24732
Fix the array-api entrypoint installation (see xfail in numpy/tests/test_public_api.py). Done in TST: enable test that checks for numpy.array_api entry point #24249.
Fix the longdouble string repr test failures on aarch64 - see Issue printing longdouble on Linux aarch64 with the Meson build #23974 and Revert "TST: disable longdouble string/print tests on Linux aarch64" #23985
Move the Pyodide CI job over to Meson
Clean up the apparently unused skip statements under [tool.cibuildwheel] in pyproject.toml; the platforms to build for are controlled in .github/workflows/wheels.yml
Compare all files included in sdist and wheel between the setup.py and Meson builds, ensure nothing went missing
Fix numpy.__version__ (see comment further down)
Remove usages of numpy.distutils in user-facing docs. Here is one example.

Note that cibuildwheel now supports Python 3.12.0b2.

setup.py and runtests.py (which uses setup.py underneath) are still used in the tools/travis-test.sh script.

setup.py and runtests.py (which uses setup.py underneath) are still used in the tools/travis-test.sh script.

Thanks, included two action items related to that, one for Travis CI and one to ensure nothing else was forgotten in any CI jobs.

Fix the xfail-ed test_cython for Windows due to a missing npyrandom.lib

This is due to the produced static library being named libnpyrandom.a rather than libnpyrandom.lib, and distutils (used by test_cython) not understanding that. For why, see https://mesonbuild.com/FAQ.html#why-does-building-my-project-with-msvc-output-static-libraries-called-libfooa.

We're planning to drop this static library completely in 2.0, and if we don't get to that we can keep the .a file extension I'd say. For 1.26.0 though, we're better off using .lib in order to not break SciPy's setup.py Windows build (that may still have to work with 1.26.0).

BLD: change file extension for installed static libraries back to `.lib`

It would be nice if the version information from versioneer could percolate up to meson at install time, like the old setuptools build. This is a little confusing:

$ pip install .
< a bunch of output elided >
Successfully installed numpy-2.0.0.dev0
$ python -c "import numpy; print(numpy.__version__)"
1.25.0rc1+265.g619a30d9a

The exact same thing (including git hash in wheel filename and in .dist-info/METADATA) is not really possible, there's no mechanism for propagating such dynamic info back up to meson-python. __version__ needs fixing though, it should be 2.0.0.dev0+xxx. I added a task to the issue description.

However, I'm also (yet again) wondering what on earth versioneer is doing here. The most recent tag is v1.25.0, but in the root of a clean repo:

>>> import versioneer
>>> versioneer.get_version()
'1.25.0rc1+264.gd38fc02fa'

Each time I look at versioneer I get a little annoyed, because it's ~3,000 lines of essentially garbage incomprehensible code split across versioneer.py and numpy/_version.py for doing exactly the same job as the ~120 LoC in https://github.com/scipy/scipy/blob/main/tools/version_utils.py.

I vote we get rid of it, and do exactly what SciPy does here - both lowest effort and a good cleanup.

RAPIDS is also going away from versioneer use as I understand it and I think many projects are. Happy if we do also.

There is a missing tag.

I think some funny merges have taken place, messing up the history.

And I think I know what happened, investigating.

For the docs, I think a reasonable editable development build command we can suggest (following pandas):

python -m pip install -ve . --no-build-isolation --config-settings editable-verbose=true

The editable-verbose=True bit avoids a confusing situation where importing numpy can cause a numpy build to happen unexpectedly, so the user will at least get output from meson in that case instead of a hanging python process.

In addition, we should document how to pass arguments via meson-python to meson, e.g. here's how to create a debug build and override the default release build:

python -m pip install . --config-settings setup-args="-Dbuildtype=debug"

Just commenting here with that for now because it took 15 minutes or so to figure out the proper way to pass arguments from pip into meson-python and then into meson.

@ngoldbaum

Good points @ngoldbaum. Do you find the editable-verbose=true useful/necessary? I don't have much experience with it yet; happy with either choice there.

Also, to save a bit of typing, --config-settings can be abbreviated now with an up-to-date pip to -C:

python -m pip install -ve . --no-build-isolation -Ceditable-verbose=true
python -m pip install . -Csetup-args="-Dbuildtype=debug"

I'm most concerned about the einsum failures using clang.

I've opened gh-24732 as a separate issue for that. It is a lot lower-prio to me than the other two items, and I don't have the hardware needed to reproduce the problem, so I'll leave that to whoever feels like digging in.

I just noticed there are still some spots in the docs that tell users to use numpy.distutils so I added a task to the issue to deal with those.

I was helping a colleague to install a numpy debug build from source yesterday, and we ran into a few issues / confusions with the new meson / spin setup. Summarizing our experience here, in case that has some useful pointers for where the docs can be improved:

I had an old numpy clone that I updated, so as a starter I ran into DOC: install from git instructions should mention submodules initialization #25132
We started from https://numpy.org/devdocs/user/building.html#building-from-source (which was pointed to from https://numpy.org/devdocs/dev/development_environment.html#building-from-source). The basic pip install instructions there work fine to get an install, except that our goal was a debug build. But it has a note on "Advanced building with Meson", which points to the SciPy docs for more information (https://docs.scipy.org/doc/scipy/building/compilers_and_options.html). However, I think those instructions from SciPy might no longer be valid for numpy? (for example my understanding is that the meson setup doesn't work right now for numpy because you vendor meson?)
And so the meson setup build --buildtype debug mentioned there didn't work to get a debug build through pip install
Going back to the original doc page, there is a section about debugging (https://numpy.org/devdocs/dev/development_environment.html#debugging), which mentions how to build numpy in debug mode using spin. First note here is that this full page doesn't mention what spin actually is. It might be explained elsewhere in the docs, but this page is the first one (after git) in the "Development" section of the docs. For example mentioning that spin is something you need to install separately (pip install spin) might be useful (it could also have been a tool included in the numpy repo itself, or similar to a makefile)
We succeeded to get a debug build with the spin build -- -Dbuildtype=debug .. command mentioned there, but what was unclear to me is what this exactly does. Did it only build numpy, or did it also install it in my dev env? My understanding now is that this essentially only builds numpy and "installs" it in a specific directory, but not in your environment, so you need to specifically add this directory on the path to use this build of numpy. This is what spin test .., spin gdb .. etc do for you (from seeing the console output doing an export PYTHONPATH= ..). This is fine if you use those tools, but in our case we wanted to install numpy into our pyarrow dev environment (copying that export command manually worked to get pyarrow rebuilt). It's not clear to me if spin also has the option to do an actual install into your environment.
Using spin gdb worked fine for me, my colleague had to use lldb. But that's without spin, so that means you need to manually set the python path to pick up the numpy build from spin build? (which is not mentioned in the docs)

In the end, once we got the gdb running with the numpy debug build, it pointed us directly to the information we needed to know, so a good ending to the story ;)

I assume that for our case, it might have been easiest to just stick with the initial pip install we tried, and I noticed that higher up here in this issue, it is mentioned how you can pass the debug build flag through pip (#23981 (comment)) with python -m pip install . --config-settings setup-args="-Dbuildtype=debug". I agree that it would be nice to document this more clearly.

@jorisvandenbossche

Thanks for sharing your experience @jorisvandenbossche. Sorry it was painful - I guess we should update these docs asap.

This is fine if you use those tools, but in our case we wanted to install numpy into our pyarrow dev environment (copying that export command manually worked to get pyarrow rebuilt). It's not clear to me if spin also has the option to do an actual install into your environment.

spin doesn't have that option, it is only useful and meant for working on numpy itself. We need to clearly separate "development work on numpy" from "building and installing numpy".

I assume that for our case, it might have been easiest to just stick with the initial pip install we tried, and I noticed that higher up here in this issue, it is mentioned how you can pass the debug build flag through pip (#23981 (comment)) with python -m pip install . --config-settings setup-args="-Dbuildtype=debug". I agree that it would be nice to document this more clearly.

Yes indeed, that should be all that is needed here. And with an up-to-date version of pip, it gets even shorted (the -C alias for --config-settings was added):

pip install . -Csetup-args=-Dbuildtype=debug

Wrt potential spin improvements:

We just released 0.8 with an lldb command
We added an install command, via pip. I can update its documentation to show how to pass in the debug flag.

This is basically done and the tracking issue has served its purpose. I'm sure there's little more loose ends somewhere, but those should get their own issues by now.

Remove usages of numpy.distutils in user-facing docs. Here is one example.

I'll open a separate issue for this, that's a doc task that is more related to the deprecation of numpy.distutils than to the introduction of Meson - it's just actionable now. EDIT: done in gh-25597

Navigation Menu

Repository navigation

Tracking issue: defaulting to Meson - follow-up tasks #23981

36 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Tracking issue: defaulting to Meson - follow-up tasks #23981

Description

Activity

charris commented on Jun 18, 2023

mattip commented on Jun 18, 2023

rgommers commented on Jun 18, 2023

rgommers commented on Jun 19, 2023

ngoldbaum commented on Jun 19, 2023

rgommers commented on Jun 19, 2023

seberg commented on Jun 19, 2023

charris commented on Jun 19, 2023

charris commented on Jun 19, 2023

charris commented on Jun 19, 2023

ngoldbaum commented on Jun 19, 2023

rgommers commented on Jun 19, 2023

36 remaining items

rgommers commented on Sep 17, 2023

ngoldbaum commented on Sep 21, 2023

jorisvandenbossche commented on Nov 16, 2023

rgommers commented on Nov 16, 2023

stefanv commented on Nov 16, 2023

rgommers commented on Jan 16, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions