Repository navigation
Pass the calling python version to meson #51
Description
Some Linux distributions encourage users to install more than one python version. For example, RHEL 8 has python3.6 as "system python", but also provides python3.9 in their repository.
When building a python extension via mesons extension_module, it is crucial to provide the calling python version to pymod.find_installation. Otherwise, meson will always use the first version in finds.
Note that, since meson is called as a subprocess in mesonpy, meson itself is not necessarily called with the same python version as pip/build.
One fix would be to pass an option, for example -Dbuild_python=python3.9, to meson that indicates which version called pip/build.
That way one can use
pymod = import('python')
python = pymod.find_installation(get_option('build_python'), ...)
in meson.build.
Activity
rgommers commentedon May 9, 2022
Thanks for opening the issue @llohse. This indeed seems to be a bit of an issue in general, however a
meson-python-specific option to work around the problem doesn't seem desirable to me. That way you get different behavior for using Meson directly (e.g.,meson setup...meson install) vs. usingmeson-python.Another way to look at this is that
meson-pythonis only for sdists and wheels. And for that use case, addingmesonas a build dependency inpyproject.tomlalready must be done anyway.Back to Meson itself, and the ways to ensure you get the right Python version:
meson.buildfile to use the Python version you want like so: `py_mod = import('python-3.9'), I don't think there are any good options. That said, you don't want to be developing with a distro-installed Python outside of a virtualenv anyway, that's inherently fragile. So I think the "fix" here is documenting "don't do that!".llohse commentedon May 9, 2022
Thanks for your comment @rgommers.
Omitting the version in
find_installationand installing meson into the build environment sounds like a good idea. I'll verify this works for me.FFY00 commentedon May 9, 2022
Hi @llohse, thank you for opening this issue. We currently force Meson to use the Python interpreter that was used to invoke the build backend by passing a cross file. This should make sure Meson is actually using the correct Python interpreter to build the module.
Can you provide some example where this appears to be broken, or problematic?
llohse commentedon May 9, 2022
Thanks for the clarification. This indeed sounds like an elegant solution. My assumption that Meson had no way of knowing the correct python version turns out to be invalid, then.
In order for this to work, one must not pass a path to
python.find_installationinmeson.build. (Would be nice to document this.)FFY00 commentedon May 9, 2022
Yes, that's true. We should definitely document this.
bnavigator commentedon Sep 10, 2022
Hi,
I might have a use case, where this either seems to not work or is incorrectly covered in the tests:
I want to migrate the openSUSE RPM package of SciPy from the deprecated distutils build to the new meson build. Instead of calling SciPy's
dev.pyormesondirectly, I wanted to give the meson-python PEP517 approach a try and found out that there is currently no openSUSE RPM package for it. So I started to packagepython-meson-pythonas well. Now openSUSE Tumbleweed is a bit special in that we package python packages for multiple Python versions. So there should be apython38-meson-python, apython39-meson-pythonand apython310-meson-pythonwhich have the files for the respecive sitelibs. However, meson itself is, as of now, only available as a single command and installed into the python3.10 sitelib. The system-wide/usr/bin/python3/is hardcoded to Python3.10. This results in the following failure of the test suite:Note that despite being called from Python 3.9 Pytest, meson finds Python 3.10 and fails to create wheels or links to Python 3.9
If this is just a quirk of the tests, I am happy to skip them. However, unless there is a guarantee for ABI3 or pure wheels, we need a way to build wheels for every Python version.
eli-schwartz commentedon Sep 11, 2022
@bnavigator,
python39-meson-python should be configuring the
mesonprogram to override the python binary to whatever is sys.executable of python39-meson-python itself.In this case, that should be
import venv; venv.create(tempdir); run(tempdir / 'bin' / 'python', '-m', 'pip', ...), so whatever pytest's sys.executable is should be used as the basis of the venv.bnavigator commentedon Sep 11, 2022
That's the thing.
meson-pythonassumesmesonbeing provided as command in the venv without checking that it is the case. Especially during the test suite.eli-schwartz commentedon Sep 11, 2022
Why does it matter which version of the command line utility
mesonis found in PATH? One of the command line arguments to themesonutility (wherever it may be found) is a cfg file specifying what python executable mesonpy (this project) would likemesonto use instead of looking for one.Again: this may be a mesonpy bug, or just a project testsuite bug, I don't know (I haven't diagnosed it) but it's definitely not a "the wrong Meson in PATH was used".
Meson is fully capable of being run with a py3.10 shebang but using a machine config file (
--native-file) pointing at 3.9 instead, and thus building code for 3.9. In theory, mesonpy already does exactly this, hardcoding the version of python that mesonpy uses. The vehicle it uses for doing this is mesonpy's sys.executable. I cannot see how pytest might mess that up, but it's even less likely that pip would mess that up, so if I had to choose something to suspect and take a closer look at, I would look at the tests.bnavigator commentedon Sep 12, 2022
Could you please point me to the command line argument? And additionally, is there any way to tell meson-python to use it? I can't find anything documented.
--python.install-envis not it.Specifically there is this:
https://mesonbuild.com/Python-module.html#find_installation
So short of patching the
meson.buildof a package frompymod.find_installation()topymod.find_installation('/my/own/custom/python3.9'), amesonfrom/usr/bin/mesonwill always find the/usr/bin/python3of the entry-point shebang. I don't say this is wrong or a bug (except for the test suite maybe), but I am missing the functionality to override this unexpected behavior.eli-schwartz commentedon Sep 12, 2022
https://github.com/FFY00/meson-python/issues/51#issuecomment-1121193585
https://github.com/FFY00/meson-python/blob/2510cde44342f3167c44ea05d650c8a524a43879/mesonpy/__init__.py#L615-L618
https://github.com/FFY00/meson-python/blob/2510cde44342f3167c44ea05d650c8a524a43879/mesonpy/__init__.py#L665-L667
https://mesonbuild.com/Machine-files.html#binaries
The binaries section of a machine file (as passed by mesonpy) will be used internally by the
find_installation()method as well....
Again, your version of Meson (and/or what version of python it's running with) is irrelevant. The only question is whether the internal mesonpy functionality of passing that native file, is passing the correct location:
I would probably add debug prints to mesonpy to see what data it is writing into that native file.
bnavigator commentedon Sep 12, 2022
It obviously does not work as you describe.
bnavigator commentedon Sep 12, 2022
Neither one. Pytest, see above. "Real world": meson-python's own build:
eli-schwartz commentedon Sep 12, 2022
While here, I noticed:
... this is a typo, the fix is https://github.com/FFY00/meson-python/pull/137
Can you test whether that fixes openSUSE? It's indeed a bug in mesonpy (and not Meson), the functionality to override the python installation to build against, just didn't work.
bnavigator commentedon Sep 12, 2022
Confirmed! Thanks!
Although I must admit that I find the whole override mechanism to be a bit fragile. The
pythonbinary is not documented in https://mesonbuild.com/Machine-files.html#binaries or in https://mesonbuild.com/Python-module.html#find_installation that it uses thepythonname. Note the 'python3' string in Meson's report when not providing the binary entry in the native file.dnicolodi commentedon Nov 24, 2022
The only actionable issue here is to document the requirements on how
import('python').find_installation()is called. I moved that into #233.ENH: Specify python version in extbuild.
MAINT: testing: specify python executable to use in extbuild (#28183)