Repository navigation
Tracking issue: defaulting to Meson - follow-up tasks #23981
Description
Activity
Note that cibuildwheel now supports Python 3.12.0b2.
Reacted by Ralf Gommerssetup.pyandruntests.py(which usessetup.pyunderneath) are still used in thetools/travis-test.shscript.setup.pyandruntests.py(which usessetup.pyunderneath) are still used in thetools/travis-test.shscript.Thanks, included two action items related to that, one for Travis CI and one to ensure nothing else was forgotten in any CI jobs.
Fix the
xfail-edtest_cythonfor Windows due to a missingnpyrandom.libThis is due to the produced static library being named
libnpyrandom.arather thanlibnpyrandom.lib, anddistutils(used bytest_cython) not understanding that. For why, see https://mesonbuild.com/FAQ.html#why-does-building-my-project-with-msvc-output-static-libraries-called-libfooa.We're planning to drop this static library completely in 2.0, and if we don't get to that we can keep the
.afile extension I'd say. For 1.26.0 though, we're better off using.libin order to not break SciPy'ssetup.pyWindows build (that may still have to work with 1.26.0).- added a commit that references this issue
on Jun 19, 2023 It would be nice if the version information from versioneer could percolate up to meson at install time, like the old setuptools build. This is a little confusing:
$ pip install . < a bunch of output elided > Successfully installed numpy-2.0.0.dev0 $ python -c "import numpy; print(numpy.__version__)" 1.25.0rc1+265.g619a30d9aReacted by Ralf GommersThe exact same thing (including git hash in wheel filename and in
.dist-info/METADATA) is not really possible, there's no mechanism for propagating such dynamic info back up tomeson-python.__version__needs fixing though, it should be2.0.0.dev0+xxx. I added a task to the issue description.However, I'm also (yet again) wondering what on earth
versioneeris doing here. The most recent tag isv1.25.0, but in the root of a clean repo:>>> import versioneer >>> versioneer.get_version() '1.25.0rc1+264.gd38fc02fa'
Each time I look at
versioneerI get a little annoyed, because it's ~3,000 lines of essentially garbage incomprehensible code split acrossversioneer.pyandnumpy/_version.pyfor doing exactly the same job as the ~120 LoC in https://github.com/scipy/scipy/blob/main/tools/version_utils.py.I vote we get rid of it, and do exactly what SciPy does here - both lowest effort and a good cleanup.
RAPIDS is also going away from versioneer use as I understand it and I think many projects are. Happy if we do also.
There is a missing tag.
I think some funny merges have taken place, messing up the history.
And I think I know what happened, investigating.
Reacted by Ralf GommersFor the docs, I think a reasonable editable development build command we can suggest (following pandas):
python -m pip install -ve . --no-build-isolation --config-settings editable-verbose=trueThe
editable-verbose=Truebit avoids a confusing situation where importing numpy can cause a numpy build to happen unexpectedly, so the user will at least get output from meson in that case instead of a hanging python process.In addition, we should document how to pass arguments via meson-python to meson, e.g. here's how to create a debug build and override the default release build:
python -m pip install . --config-settings setup-args="-Dbuildtype=debug"Just commenting here with that for now because it took 15 minutes or so to figure out the proper way to pass arguments from pip into meson-python and then into meson.
Reacted by Ralf GommersGood points @ngoldbaum. Do you find the
editable-verbose=trueuseful/necessary? I don't have much experience with it yet; happy with either choice there.Also, to save a bit of typing,
--config-settingscan be abbreviated now with an up-to-datepipto-C:python -m pip install -ve . --no-build-isolation -Ceditable-verbose=true python -m pip install . -Csetup-args="-Dbuildtype=debug"Reacted by Nathan Goldbaum36 remaining items
I'm most concerned about the einsum failures using clang.
I've opened gh-24732 as a separate issue for that. It is a lot lower-prio to me than the other two items, and I don't have the hardware needed to reproduce the problem, so I'll leave that to whoever feels like digging in.
I just noticed there are still some spots in the docs that tell users to use
numpy.distutilsso I added a task to the issue to deal with those.Reacted by Ralf GommersI was helping a colleague to install a numpy debug build from source yesterday, and we ran into a few issues / confusions with the new meson / spin setup. Summarizing our experience here, in case that has some useful pointers for where the docs can be improved:
- I had an old numpy clone that I updated, so as a starter I ran into DOC: install from git instructions should mention submodules initialization #25132
- We started from https://numpy.org/devdocs/user/building.html#building-from-source (which was pointed to from https://numpy.org/devdocs/dev/development_environment.html#building-from-source). The basic
pip installinstructions there work fine to get an install, except that our goal was a debug build. But it has a note on "Advanced building with Meson", which points to the SciPy docs for more information (https://docs.scipy.org/doc/scipy/building/compilers_and_options.html). However, I think those instructions from SciPy might no longer be valid for numpy? (for example my understanding is that themeson setupdoesn't work right now for numpy because you vendor meson?)
And so themeson setup build --buildtype debugmentioned there didn't work to get a debug build throughpip install - Going back to the original doc page, there is a section about debugging (https://numpy.org/devdocs/dev/development_environment.html#debugging), which mentions how to build numpy in debug mode using
spin. First note here is that this full page doesn't mention whatspinactually is. It might be explained elsewhere in the docs, but this page is the first one (after git) in the "Development" section of the docs. For example mentioning thatspinis something you need to install separately (pip install spin) might be useful (it could also have been a tool included in the numpy repo itself, or similar to a makefile) - We succeeded to get a debug build with the
spin build -- -Dbuildtype=debug ..command mentioned there, but what was unclear to me is what this exactly does. Did it only build numpy, or did it also install it in my dev env? My understanding now is that this essentially only builds numpy and "installs" it in a specific directory, but not in your environment, so you need to specifically add this directory on the path to use this build of numpy. This is whatspin test ..,spin gdb ..etc do for you (from seeing the console output doing anexport PYTHONPATH= ..). This is fine if you use those tools, but in our case we wanted to install numpy into our pyarrow dev environment (copying thatexportcommand manually worked to get pyarrow rebuilt). It's not clear to me ifspinalso has the option to do an actual install into your environment. - Using
spin gdbworked fine for me, my colleague had to uselldb. But that's withoutspin, so that means you need to manually set the python path to pick up the numpy build fromspin build? (which is not mentioned in the docs)
In the end, once we got the gdb running with the numpy debug build, it pointed us directly to the information we needed to know, so a good ending to the story ;)
I assume that for our case, it might have been easiest to just stick with the initial
pip installwe tried, and I noticed that higher up here in this issue, it is mentioned how you can pass the debug build flag through pip (#23981 (comment)) withpython -m pip install . --config-settings setup-args="-Dbuildtype=debug". I agree that it would be nice to document this more clearly.Reacted by Alenka Frim and Melissa Weber MendonçaThanks for sharing your experience @jorisvandenbossche. Sorry it was painful - I guess we should update these docs asap.
This is fine if you use those tools, but in our case we wanted to install numpy into our pyarrow dev environment (copying that
exportcommand manually worked to get pyarrow rebuilt). It's not clear to me ifspinalso has the option to do an actual install into your environment.spindoesn't have that option, it is only useful and meant for working onnumpyitself. We need to clearly separate "development work on numpy" from "building and installing numpy".I assume that for our case, it might have been easiest to just stick with the initial
pip installwe tried, and I noticed that higher up here in this issue, it is mentioned how you can pass the debug build flag through pip (#23981 (comment)) withpython -m pip install . --config-settings setup-args="-Dbuildtype=debug". I agree that it would be nice to document this more clearly.Yes indeed, that should be all that is needed here. And with an up-to-date version of
pip, it gets even shorted (the-Calias for--config-settingswas added):pip install . -Csetup-args=-Dbuildtype=debugReacted by Melissa Weber Mendonça and Joris Van den BosscheWrt potential spin improvements:
- We just released 0.8 with an lldb command
- We added an install command, via pip. I can update its documentation to show how to pass in the debug flag.
Reacted by Sebastian Berg, Juan Luis Cano Rodríguez, Melissa Weber Mendonça and Rohit GoswamiThis is basically done and the tracking issue has served its purpose. I'm sure there's little more loose ends somewhere, but those should get their own issues by now.
Remove usages of
numpy.distutilsin user-facing docs. Here is one example.I'll open a separate issue for this, that's a doc task that is more related to the deprecation of
numpy.distutilsthan to the introduction of Meson - it's just actionable now. EDIT: done in gh-25597
After gh-23838 we now default to building with Meson (via
meson-python, when invoking a build withpip install .or similar) inmain. There are a number of more niche build configs not yet ported, and a few things werexfail-ed and need fixing up. Here is a list, roughly in order of priority:CFLAGS/CXXFLAGS- I'll work on this next): see BLD: improve handling of CBLAS, add-Duse-ilp64build option #23984NPY_BLAS_ORDER). Done in ENH: meson: implement BLAS/LAPACK auto-detection and many CI jobs #24893pypy3.9CI job on Azure. Done in BLD/CI: re-enable ILP64 usage and PyPy job in Azure #24238.runtests.py)long doublerepresentation check so the builds works on some of the more niche platforms again (xref BUG: New meson system fails to identify long double representation on 64-bit BE platforms #23972)xfail-edtest_cythonfor Windows due to a missingnpyrandom.lib(done in BLD: change file extension for installed static libraries back to.lib#23983)runtests.pyandsetup.py, then running CI and ensuring nothing fails). Done - there were a lot left. See Restructuring GitHub Actions CI jobs #24410 for the tracking issue for CI migration.bin/f2py3XCLI install, in addition to thef2pyandf2py3ones (see project.scriptsinpyproject.toml`, and commit f22a33b). Or can/should that interface be dropped? Multiple installs into a single env is considered bad practice by now. Dropped in BLD: only install thef2pycommand, notf2py3orf2py3.X#24235.einsumtest failures withint8/uint8that were xfailed in BLD: switch to meson-python as the default build backend #23838 (EDIT: these two tests fail withclang-clon Windows too, see ENH: Add changes that allow NumPy to compile with clang-cl #20866 (comment)). EDIT: opened separate issue: BUG:einsumwith int8/uint8 input is buggy #24732xfailinnumpy/tests/test_public_api.py). Done in TST: enable test that checks fornumpy.array_apientry point #24249.longdoublestring repr test failures onaarch64- see Issue printinglongdoubleon Linux aarch64 with the Meson build #23974 and Revert "TST: disable longdouble string/print tests on Linux aarch64" #23985skipstatements under[tool.cibuildwheel]inpyproject.toml; the platforms to build for are controlled in.github/workflows/wheels.ymlnumpy.__version__(see comment further down)numpy.distutilsin user-facing docs. Here is one example.