pyodide-build: unable to locate numpy package while building the pymatgen package

🐛 Bug

During the build of a pymatgen wasm/emscripten whl file following the tutorial: https://pyodide.org/en/stable/development/building-and-testing-packages.html, the 'pyodide build' command returns an error: "ModuleNotFoundError: No module named 'numpy'

To Reproduce

git clone https://github.com/materialsproject/pymatgen

then follow this tutorial

Expected behavior

build pymatgen-cp310-cp310-emscripten_3_1_27_wasm32.whl file

Environment

Pyodide Version: 0.2.3
Browser version: 120.0.6099.71
Any other relevant information: Pymatgen version: 2023.11.18

Error Log

xbuild environment already exists, skipping download                                                                                                                         
Installing xbuild environment                                                                                                                                                
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (Cython>=0.29.23, setuptools>=65.0.0, versioneer[toml], wheel)
* Getting dependencies for wheel...
Traceback (most recent call last):
  File "/home/jvanbuskirk/.local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 351, in <module>
    main()
  File "/home/jvanbuskirk/.local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 333, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/home/jvanbuskirk/.local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
    return hook(config_settings)
  File "/tmp/build-env-bmttes29/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=['wheel'])
  File "/tmp/build-env-bmttes29/lib/python3.10/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
    self.run_setup()
  File "/tmp/build-env-bmttes29/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
    exec(code, locals())
  File "<string>", line 8, in <module>
ModuleNotFoundError: No module named 'numpy'
* Getting dependencies for wheel...
Traceback (most recent call last):
  File "/home/jvanbuskirk/.local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 351, in <module>
    main()
  File "/home/jvanbuskirk/.local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 333, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/home/jvanbuskirk/.local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
    return hook(config_settings)
  File "/tmp/build-env-bmttes29/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=['wheel'])
  File "/tmp/build-env-bmttes29/lib/python3.10/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
    self.run_setup()
  File "/tmp/build-env-bmttes29/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
    exec(code, locals())
  File "<string>", line 8, in <module>
ModuleNotFoundError: No module named 'numpy'

ERROR Backend subproccess exited when trying to invoke get_requires_for_build_wheel

Additional context

As stated in the tutorial, I verified that I can build the whl file with the regular

pip install build
python -m build

workflow, and indeed, that resulted in a whl file appropriate for my local operating system (Ubuntu 22.04.3, x86_64)

This issue seems to be related to an existing issue: #4160 but pymatgen comes with a pyproject.toml, which requires setuptools, not meson for building. In the other reported issue, pandas appeared to give a similar error when compilation occured with setuptools, but in pandas 2.1.0, the pyproject.toml uses meson, and deos not give the error. Is it possible the error is related to the use of setuptools?

I am getting the same error when building moleculekit
If you look at the pyproject.toml it includes numpy in the build dependencies

[build-system]
requires = [
  "setuptools",
  "numpy>=1.18.5",
  "Cython>=0.29.21",
  "toml",
  "versioneer[toml]==0.28",
]
build-backend = "setuptools.build_meta"

but when running pyodide-build it removes it from the packages

Downloading xbuild environment                                                                                                          
Installing xbuild environment                                                                                                           
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (Cython>=0.29.21, setuptools, toml, versioneer[toml]==0.28)

I played around to see if it was maybe reading these packages from elsewhere and removed versioneer from the list and indeed pyodide-build updated that message, so it's reading them from pyproject.toml but for some reason it ignores numpy when it's in that list.

python -m build works perfectly fine

I dug into the source code.
In the file pypabuild.py which is the modified pypa/build used by pyodide-build, there is a function remove_avoided_requirements which takes as input get_unisolated_packages() + AVOIDED_REQUIREMENTS.
Looking at build_env.py in the get_unisolated_packages() it reads a file /home/user/moleculekit/.pyodide-xbuildenv/xbuildenv/pyodide-root/unisolated.txt which contains numpy in it. So indeed pyodide-build is removing on purpose numpy from our build requirements. I don't understand why though.

Removing numpy from the unisolated.txt file continues along with the build but eventually crashes with:

AssertionError: would build wheel with unsupported tag ('cp310', 'cp311', 'emscripten_3_1_45_wasm32')

ERROR Backend subproccess exited when trying to invoke build_wheel

Which according to this stackoverflow question from 2 years ago is a bug in python 3.10 and numpy? https://stackoverflow.com/a/66908804
I'm out of ideas

Hi, Stefan, thanks for sharing your similar error. Hopefully this is addressed soon. As an update, I did manage to make a little more progress.

Since you were able to determine the error is confined to python3.10 and numpy, I went ahead and installed a local copy of python3.11, and all of the requisite packages, hoping to simply repeat the process with the newer python version.

Unfortunately, pymatgen does not have a /home/user/pymatgen/.pyodide-xbuildenv/ directory, but I was eventually able to get it running on pyodide anyways by hosting a local copy of the .tar.gz file from https://pypi.org/project/pymatgen/#files following this procedure: https://pyodide.org/en/stable/usage/loading-custom-python-code.html#from-python. Note, if you do this, you'll need to extract the [package] directory from the untared [package]-X.X.X directory, then retar that into the .tar.gz you use to upload.

This worked for pymatgen, but then I had to go through and find/install all of it's dependencies manually. This led me to needing to repeat the process for another unsupported package, spglib. This package has a .pyodide-buildenv/ directory though. Following your procedure of removing numpy from the /home/user/package/.pyodide-xbuildenv/xbuildenv/pyodide-root/unisolated.txt file, I was able to call pyodide build in the /home/user/package directory, and build a -none-any.whl file.

Unfortunately, now this wheel file is being installed correctly, but is not being seen by pymatgen. I can call micropip.install("./spglib-2.2.0-py3-none-any.whl");, and importing micropip and calling micropip.list() shows spglib is imported, but I still get ModuleNotFoundError: No module named 'spglib' when one of the pymatgen modules calls import spglib.

Does anyone have any thoughts about this?

Any update on this issue? It's still around in the latest pyodide-build version 0.25.1

Hi Stefan!
Sort of. It's been a while, so this may be slightly inaccurate, but I never got pymatgen to build with the necessary numpy dependency. Instead, I noticed that I didn't intend to use the portion of pymatgen that required numpy, so I edited my local copy of pymatgen to remove the parts that required numpy, essentially removing functionality from pymatgen. While this is a terrible practice for a plethora of reasons, it did work...
Best of luck!

@ryanking13

I guess we should look into this @ryanking13...

@hoodmane

Oh, thanks for the ping @hoodmane. I must have missed this issue.

In the file pypabuild.py which is the modified pypa/build used by pyodide-build, there is a function remove_avoided_requirements which takes as input get_unisolated_packages() + AVOIDED_REQUIREMENTS.
Looking at build_env.py in the get_unisolated_packages() it reads a file /home/user/moleculekit/.pyodide-xbuildenv/xbuildenv/pyodide-root/unisolated.txt which contains numpy in it. So indeed pyodide-build is removing on purpose numpy from our build requirements. I don't understand why though.

This is intended. WebAssembly is a 32-bit environment, so if you install numpy from PyPI when you build, some headers will be incorrect because their values are set based on a 64-bit system. So instead of pulling numpy from PyPI, we redirect it to the numpy we built for WebAssembly to handle it, but I think something was going wrong and this wasn't working properly.

@ryanking13

@ryanking13 I see, but the issue is that my package contains some cython modules and that cython code uses numpy types so we need to have numpy installed to be able to compile the cython code. This used to work in the past as I was able to correctly build and use my packages through pyodide including the cython code.

@stefdoerr

@stefdoerr Are you building with Python 3.11?

I just successfully built moleculekit with the following:

git clone https://github.com/emscripten-core/emsdk --depth 1
./emsdk/emsdk install 3.1.46
source ./emsdk/emsdk_enc.sh
python3.11 -m venv .venv
source .venv/bin/activate
pip install pyodide-build
pyodide build

so it is at least possible to build pymatgen. It would help if you could give more precise information about what you did for us to reproduce it.

Ah from your message above:

would build wheel with unsupported tag ('cp310', 'cp311', 'emscripten_3_1_45_wasm32')

it sounds like you are trying to build with Python3.10 but you need to use Python 3.11. I think that'll fix your trouble. We should have better error reporting for this...

Oh well.. that fixed it 😄 Thanks for the help! But yes that error is really not very helpful, I thought it meant that it had trouble with both python 3.10 and 3.11 not that I should switch from 3.10 to 3.11.

Navigation Menu

Repository navigation

pyodide-build: unable to locate numpy package while building the pymatgen package #4347

🐛 Bug

To Reproduce

Expected behavior

Environment

Error Log

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

pyodide-build: unable to locate numpy package while building the pymatgen package #4347

Description

🐛 Bug

To Reproduce

Expected behavior

Environment

Error Log

Additional context

Activity

stefdoerr commented on Dec 20, 2023

stefdoerr commented on Dec 20, 2023

stefdoerr commented on Dec 20, 2023

jonvanbuskirk0 commented on Dec 21, 2023

stefdoerr commented on Apr 26, 2024

jonvanbuskirk0 commented on Apr 30, 2024

hoodmane commented on Apr 30, 2024

ryanking13 commented on May 1, 2024

stefdoerr commented on May 6, 2024

hoodmane commented on May 6, 2024

hoodmane commented on May 6, 2024

hoodmane commented on May 6, 2024

stefdoerr commented on May 7, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions