BUG: Cannot build main on MacOS M1 #24450
Description
Activity
Have you tried using
spin build --clean? It seems that the configuration file for the_umath_tests.dispatch.htarget is empty for some reason. Though I'm not certain why this occurred, could it be due to configuration process was interpreted during certain stage.Could you provide the Meson log? You can find it at
build/meson-logs/meson-log.txt. Sharing the generated config file atbuild/numpy/core/_umath_tests.dispatch.hwould also help me better understand the issue. Thanks!- addedMesonItems related to the introduction of Meson as the new build system for NumPyItems related to the introduction of Meson as the new build system for NumPy
on Aug 18, 2023 Have you tried using spin build --clean?
Yes, sorry for not clarifying this. Absolutely fresh setup, starting with git/git clone has the same problem.
It seems that the configuration file for the _umath_tests.dispatch.h target is empty for some reason
$ cat build/numpy/core/_umath_tests.dispatch.h /* Autogenerated by the Meson features module. */ /* Do not edit, your changes will be lost. */ #undef NPY__MTARGETS_EXPAND #define NPY__MTARGETS_EXPAND(X) X #undef NPY_MTARGETS_CONF_BASELINE #define NPY_MTARGETS_CONF_BASELINE(EXEC_CB, ...) NPY__MTARGETS_EXPAND(EXEC_CB(__VA_ARGS__)) #undef NPY_MTARGETS_CONF_DISPATCH #define NPY_MTARGETS_CONF_DISPATCH(TEST_CB, EXEC_CB, ...) \ NPY__MTARGETS_EXPAND(EXEC_CB((TEST_CB(ASIMD)&&TEST_CB(ASIMDHP)), ASIMDHP, __VA_ARGS__))Could you provide the Meson log?
Sure, let me know if I missed something that could help
The configuration stage appears to be correct, and the auto-generated file is not empty and well-defined. However, an empty inclusion has somehow been included within the following code section:
numpy/numpy/core/src/umath/_umath_tests.dispatch.c
Lines 14 to 18 in f884cf6
#ifndef NPY_DISABLE_OPTIMIZATION #include "_umath_tests.dispatch.h" #endif NPY_CPU_DISPATCH_DECLARE(const char *_umath_tests_dispatch_func, (void)) To address this issue, it's suggested that you apply the following patch to explicitly define the generated config macros and rebuild the code to determine if the build process passes:
diff --git a/numpy/core/src/umath/_umath_tests.dispatch.c b/numpy/core/src/umath/_umath_tests.dispatch.c index 9d8df4c86..73a11a16d 100644 --- a/numpy/core/src/umath/_umath_tests.dispatch.c +++ b/numpy/core/src/umath/_umath_tests.dispatch.c @@ -13,6 +13,17 @@ #ifndef NPY_DISABLE_OPTIMIZATION #include "_umath_tests.dispatch.h" + /* Autogenerated by the Meson features module. */ + /* Do not edit, your changes will be lost. */ + #undef NPY__MTARGETS_EXPAND + #define NPY__MTARGETS_EXPAND(X) X + + #undef NPY_MTARGETS_CONF_BASELINE + #define NPY_MTARGETS_CONF_BASELINE(EXEC_CB, ...) NPY__MTARGETS_EXPAND(EXEC_CB(__VA_ARGS__)) + + #undef NPY_MTARGETS_CONF_DISPATCH + #define NPY_MTARGETS_CONF_DISPATCH(TEST_CB, EXEC_CB, ...) \ + NPY__MTARGETS_EXPAND(EXEC_CB((TEST_CB(ASIMD)&&TEST_CB(ASIMDHP)), ASIMDHP, __VA_ARGS__)) #endif NPY_CPU_DISPATCH_DECLARE(const char *_umath_tests_dispatch_func, (void))
However, before applying the patch, it would be helpful to provide the complete build log to gain better insights into the issue. You can generate the build log using the following command:
git clean -xdf spin build --clean &> numpy_spin_build.txtThen please attach the
numpy_spin_build.txtfile to assist in diagnosing the potential problem. Based on my understanding, it's possible that an empty header of_umath_tests.dispatch.hdefined else where in the searched directories might be overriding the expectedbuild/numpy/core/_umath_tests.dispatch.hor maybe not empty header but likely the disutils configuration header been generated in place through the command:
python setup.py build_ext -iso please make sure of cleaning the directory before using spin build through:
git clean -xdfor maybe not empty header but likely the disutils configuration header been generated in place through the command
Yes, this seems to be the issue.
git clean -dxdid the trick, it's working now.
I was removing build directory instead, apparently this isn't enough.Just in case, these are the files that were removed by
git clean, probably spin's--cleanflag should remove them too.Removing .hypothesis/ Removing .pytest_cache/ Removing .spin/__pycache__/ Removing .vscode/ Removing build/ Removing cythonize.dat Removing numpy/__config__.py Removing numpy/__pycache__/ Removing numpy/_build_utils/__pycache__/ Removing numpy/_pyinstaller/__pycache__/ Removing numpy/_typing/__pycache__/ Removing numpy/_utils/__pycache__/ Removing numpy/array_api/__pycache__/ Removing numpy/array_api/tests/__pycache__/ Removing numpy/compat/__pycache__/ Removing numpy/compat/tests/__pycache__/ Removing numpy/core/__pycache__/ Removing numpy/core/_multiarray_tests.cpython-311-darwin.so Removing numpy/core/_multiarray_umath.cpython-311-darwin.so Removing numpy/core/_operand_flag_tests.cpython-311-darwin.so Removing numpy/core/_rational_tests.cpython-311-darwin.so Removing numpy/core/_simd.cpython-311-darwin.so Removing numpy/core/_struct_ufunc_tests.cpython-311-darwin.so Removing numpy/core/_umath_tests.cpython-311-darwin.so Removing numpy/core/code_generators/__pycache__/ Removing numpy/core/include/numpy/__multiarray_api.c Removing numpy/core/include/numpy/__multiarray_api.h Removing numpy/core/include/numpy/__ufunc_api.c Removing numpy/core/include/numpy/__ufunc_api.h Removing numpy/core/include/numpy/__umath_generated.c Removing numpy/core/include/numpy/_numpyconfig.h Removing numpy/core/include/numpy/_umath_doc_generated.h Removing numpy/core/include/numpy/config.h Removing numpy/core/lib/ Removing numpy/core/src/_simd/_simd.dispatch.c Removing numpy/core/src/_simd/_simd.dispatch.h Removing numpy/core/src/_simd/_simd_data.inc Removing numpy/core/src/_simd/_simd_inc.h Removing numpy/core/src/common/npy_sort.h Removing numpy/core/src/common/templ_common.h Removing numpy/core/src/multiarray/_multiarray_tests.c Removing numpy/core/src/multiarray/argfunc.dispatch.c Removing numpy/core/src/multiarray/argfunc.dispatch.h Removing numpy/core/src/multiarray/arraytypes.c Removing numpy/core/src/multiarray/arraytypes.h Removing numpy/core/src/multiarray/einsum.c Removing numpy/core/src/multiarray/einsum_sumprod.c Removing numpy/core/src/multiarray/lowlevel_strided_loops.c Removing numpy/core/src/multiarray/nditer_templ.c Removing numpy/core/src/multiarray/scalartypes.c Removing numpy/core/src/npymath/ieee754.c Removing numpy/core/src/npymath/npy_math_complex.c Removing numpy/core/src/npymath/npy_math_internal.h Removing numpy/core/src/npysort/simd_qsort.dispatch.h Removing numpy/core/src/npysort/simd_qsort_16bit.dispatch.h Removing numpy/core/src/umath/_umath_tests.c Removing numpy/core/src/umath/_umath_tests.dispatch.asimdhp.c Removing numpy/core/src/umath/_umath_tests.dispatch.h Removing numpy/core/src/umath/funcs.inc Removing numpy/core/src/umath/loops.c Removing numpy/core/src/umath/loops.h Removing numpy/core/src/umath/loops_arithm_fp.dispatch.c Removing numpy/core/src/umath/loops_arithm_fp.dispatch.h Removing numpy/core/src/umath/loops_arithmetic.dispatch.c Removing numpy/core/src/umath/loops_arithmetic.dispatch.h Removing numpy/core/src/umath/loops_autovec.dispatch.c Removing numpy/core/src/umath/loops_autovec.dispatch.h Removing numpy/core/src/umath/loops_comparison.dispatch.c Removing numpy/core/src/umath/loops_comparison.dispatch.h Removing numpy/core/src/umath/loops_exponent_log.dispatch.c Removing numpy/core/src/umath/loops_exponent_log.dispatch.h Removing numpy/core/src/umath/loops_hyperbolic.dispatch.c Removing numpy/core/src/umath/loops_hyperbolic.dispatch.h Removing numpy/core/src/umath/loops_logical.dispatch.c Removing numpy/core/src/umath/loops_logical.dispatch.h Removing numpy/core/src/umath/loops_minmax.dispatch.c Removing numpy/core/src/umath/loops_minmax.dispatch.h Removing numpy/core/src/umath/loops_modulo.dispatch.c Removing numpy/core/src/umath/loops_modulo.dispatch.h Removing numpy/core/src/umath/loops_trigonometric.dispatch.c Removing numpy/core/src/umath/loops_trigonometric.dispatch.h Removing numpy/core/src/umath/loops_umath_fp.dispatch.c Removing numpy/core/src/umath/loops_umath_fp.dispatch.h Removing numpy/core/src/umath/loops_unary.dispatch.c Removing numpy/core/src/umath/loops_unary.dispatch.h Removing numpy/core/src/umath/loops_unary_complex.dispatch.c Removing numpy/core/src/umath/loops_unary_complex.dispatch.h Removing numpy/core/src/umath/loops_unary_fp.dispatch.c Removing numpy/core/src/umath/loops_unary_fp.dispatch.h Removing numpy/core/src/umath/loops_unary_fp_le.dispatch.c Removing numpy/core/src/umath/loops_unary_fp_le.dispatch.h Removing numpy/core/src/umath/loops_utils.h Removing numpy/core/src/umath/matmul.c Removing numpy/core/src/umath/matmul.h Removing numpy/core/src/umath/scalarmath.c Removing numpy/core/tests/__pycache__/ Removing numpy/distutils/__config__.py Removing numpy/distutils/__pycache__/ Removing numpy/distutils/command/__pycache__/ Removing numpy/distutils/fcompiler/__pycache__/ Removing numpy/distutils/tests/__pycache__/ Removing numpy/f2py/__pycache__/ Removing numpy/f2py/tests/__pycache__/ Removing numpy/fft/__pycache__/ Removing numpy/fft/_pocketfft_internal.cpython-311-darwin.so Removing numpy/fft/tests/__pycache__/ Removing numpy/lib/__pycache__/ Removing numpy/lib/tests/__pycache__/ Removing numpy/linalg/__pycache__/ Removing numpy/linalg/_umath_linalg.cpython-311-darwin.so Removing numpy/linalg/lapack_lite.cpython-311-darwin.so Removing numpy/linalg/tests/__pycache__/ Removing numpy/ma/__pycache__/ Removing numpy/ma/tests/__pycache__/ Removing numpy/matrixlib/__pycache__/ Removing numpy/matrixlib/tests/__pycache__/ Removing numpy/polynomial/__pycache__/ Removing numpy/polynomial/tests/__pycache__/ Removing numpy/random/__pycache__/ Removing numpy/random/_bounded_integers.c Removing numpy/random/_bounded_integers.cpython-311-darwin.so Removing numpy/random/_bounded_integers.pxd Removing numpy/random/_bounded_integers.pyx Removing numpy/random/_common.c Removing numpy/random/_common.cpython-311-darwin.so Removing numpy/random/_examples/cffi/__pycache__/ Removing numpy/random/_generator.c Removing numpy/random/_generator.cpython-311-darwin.so Removing numpy/random/_mt19937.c Removing numpy/random/_mt19937.cpython-311-darwin.so Removing numpy/random/_pcg64.c Removing numpy/random/_pcg64.cpython-311-darwin.so Removing numpy/random/_philox.c Removing numpy/random/_philox.cpython-311-darwin.so Removing numpy/random/_sfc64.c Removing numpy/random/_sfc64.cpython-311-darwin.so Removing numpy/random/bit_generator.c Removing numpy/random/bit_generator.cpython-311-darwin.so Removing numpy/random/lib/ Removing numpy/random/mtrand.c Removing numpy/random/mtrand.cpython-311-darwin.so Removing numpy/random/tests/__pycache__/ Removing numpy/random/tests/data/__pycache__/ Removing numpy/testing/__pycache__/ Removing numpy/testing/_private/__pycache__/ Removing numpy/testing/tests/__pycache__/ Removing numpy/tests/__pycache__/ Removing numpy/typing/__pycache__/ Removing numpy/typing/tests/__pycache__/ Removing numpy/version.py Removing tools/npy_tempita/__pycache__/ Removing var/@stefanv is there a way to pass a list of files to extend
spin clean?@code-of-kpp Thank you for your assistance in investigating this bug. I've updated the reproducer code within the pull request description and have also opened a pull request that fixes this issue (#24468).
@mattip, It seems the easiest fix is to ensures that distutils generates these headers in the build directory, even with the inplace option enabled
@stefanv is there a way to pass a list of files to extend
spin clean?We don't have such a command, and I don't think we should add it. It's going to be inherently unreliable just like
setup.py cleanwas not reliable. It's simply not possible to know what to delete I think, unless it's a simple alias forgit clean -xdf. So better to continue to recommend to usegit clean -xdfdirectly.Thank you all!
A couple of notes (and feel free to let me know, if they deserve an issue on their own):
- build_ext succeeded, but tests were failing in a strange way, that's when @seberg suggested to switch to spin
- git clean isn't normally an option for me, as I prefer to keep related virtual environments and IDE configuration in the same folder, if that wouldn't be the first time I cloned the repo to this laptop, I would lose a bunch of things.
Describe the issue:
Most likely related to 4ec0182 by @seiko2plus and gh-23096.
Fresh install, git clone, submodules update
dependencies installed via
pip install -r(I installed all the files, but when just build dependencies installed, the situation is the same).I'm building with openblass installed from homebrew, and I do:
export PKG_CONFIG_PATH=/opt/homebrew/Cellar/openblas/0.3.23/lib/pkgconfig/for it to get detected.
I tried to build with clang 16.0.6 from brew and 14.0.3 (coming with MacOs).
spin buildfails.Reproduce the code example:
Edit(Sayed): You have to build numpy using distutils with the in-place option enabled, followed by
spin build --clean.Error message:
Runtime information:
Context for the issue:
I was working on
numpysprint at EuroSciPy and couldn't finish the setup./cc @seberg