Repository navigation
BUG: Cannot build main on MacOS M1 #24450
Description
Describe the issue:
Most likely related to 4ec0182 by @seiko2plus and gh-23096.
Fresh install, git clone, submodules update
dependencies installed via pip install -r (I installed all the files, but when just build dependencies installed, the situation is the same).
I'm building with openblass installed from homebrew, and I do:
export PKG_CONFIG_PATH=/opt/homebrew/Cellar/openblas/0.3.23/lib/pkgconfig/
for it to get detected.
I tried to build with clang 16.0.6 from brew and 14.0.3 (coming with MacOs).
spin build fails.
Reproduce the code example:
Edit(Sayed): You have to build numpy using distutils with the in-place option enabled, followed by spin build --clean.
python setup.py build_ext -i
rm -rf ./build
spin build --cleanError message:
...
Test features "NEON NEON_FP16 NEON_VFPV4 ASIMD" : Supported
Test features "ASIMDHP" : Supported
Test features "ASIMDFHM" : Supported
Configuring npy_cpu_dispatch_config.h using configuration
Message:
CPU Optimization Options
baseline:
Requested : min
Enabled : NEON NEON_FP16 NEON_VFPV4 ASIMD
dispatch:
Requested : max -xop -fma4
Enabled : ASIMDHP ASIMDFHM
...
[44/306] Compiling C object numpy/core/lib_uma...SIMDHP.a.p/src_umath__umath_tests.dispatch.c.
FAILED: numpy/core/lib_umath_tests.dispatch.h_ASIMDHP.a.p/src_umath__umath_tests.dispatch.c.o
/usr/bin/clang -Inumpy/core/lib_umath_tests.dispatch.h_ASIMDHP.a.p -Inumpy/core -I../numpy/core -Inumpy/core/include -I../numpy/core/include -I../numpy/core/src/common -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.11/include/python3.11 -I/opt/homebrew/opt/llvm/include -I/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu -fcolor-diagnostics -Wall -Winvalid-pch -std=c99 -O2 -g -fno-strict-aliasing -ftrapping-math -DNPY_HAVE_NEON_VFPV4 -DNPY_HAVE_NEON_FP16 -DNPY_HAVE_NEON -DNPY_HAVE_ASIMD -DNPY_HAVE_ASIMDHP -march=armv8.2-a+fp16 -DNPY_MTARGETS_CURRENT=ASIMDHP -MD -MQ numpy/core/lib_umath_tests.dispatch.h_ASIMDHP.a.p/src_umath__umath_tests.dispatch.c.o -MF numpy/core/lib_umath_tests.dispatch.h_ASIMDHP.a.p/src_umath__umath_tests.dispatch.c.o.d -o numpy/core/lib_umath_tests.dispatch.h_ASIMDHP.a.p/src_umath__umath_tests.dispatch.c.o -c ../numpy/core/src/umath/_umath_tests.dispatch.c
../numpy/core/src/umath/_umath_tests.dispatch.c:18:1: error: expected identifier or '('
NPY_CPU_DISPATCH_DECLARE(const char *_umath_tests_dispatch_func, (void))
^
/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu/npy_cpu_dispatch_config.h:194:5: note: expanded from macro 'NPY_CPU_DISPATCH_DECLARE'
NPY_MTARGETS_CONF_DISPATCH(NPY_CPU_DISPATCH_DECLARE_CHK_, NPY_CPU_DISPATCH_DECLARE_CB_, __VA_ARGS__) \
^
/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu/npy_cpu_dispatch_config.h:94:10: note: expanded from macro 'NPY_MTARGETS_CONF_DISPATCH'
&&"Expected config header that generated by mod_features.multi_targets()";
^
../numpy/core/src/umath/_umath_tests.dispatch.c:18:1: error: expected identifier or '('
/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu/npy_cpu_dispatch_config.h:195:5: note: expanded from macro 'NPY_CPU_DISPATCH_DECLARE'
NPY_MTARGETS_CONF_BASELINE(NPY_CPU_DISPATCH_DECLARE_BASE_CB_, __VA_ARGS__)
^
/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu/npy_cpu_dispatch_config.h:92:10: note: expanded from macro 'NPY_MTARGETS_CONF_BASELINE'
&&"Expected config header that generated by mod_features.multi_targets()";
^
../numpy/core/src/umath/_umath_tests.dispatch.c:19:1: error: expected identifier or '('
NPY_CPU_DISPATCH_DECLARE(extern const char *_umath_tests_dispatch_var)
^
/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu/npy_cpu_dispatch_config.h:194:5: note: expanded from macro 'NPY_CPU_DISPATCH_DECLARE'
NPY_MTARGETS_CONF_DISPATCH(NPY_CPU_DISPATCH_DECLARE_CHK_, NPY_CPU_DISPATCH_DECLARE_CB_, __VA_ARGS__) \
^
/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu/npy_cpu_dispatch_config.h:94:10: note: expanded from macro 'NPY_MTARGETS_CONF_DISPATCH'
&&"Expected config header that generated by mod_features.multi_targets()";
^
../numpy/core/src/umath/_umath_tests.dispatch.c:19:1: error: expected identifier or '('
/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu/npy_cpu_dispatch_config.h:195:5: note: expanded from macro 'NPY_CPU_DISPATCH_DECLARE'
NPY_MTARGETS_CONF_BASELINE(NPY_CPU_DISPATCH_DECLARE_BASE_CB_, __VA_ARGS__)
^
/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu/npy_cpu_dispatch_config.h:92:10: note: expanded from macro 'NPY_MTARGETS_CONF_BASELINE'
&&"Expected config header that generated by mod_features.multi_targets()";
^
../numpy/core/src/umath/_umath_tests.dispatch.c:20:1: error: expected identifier or '('
NPY_CPU_DISPATCH_DECLARE(void _umath_tests_dispatch_attach, (PyObject *list))
^
/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu/npy_cpu_dispatch_config.h:194:5: note: expanded from macro 'NPY_CPU_DISPATCH_DECLARE'
NPY_MTARGETS_CONF_DISPATCH(NPY_CPU_DISPATCH_DECLARE_CHK_, NPY_CPU_DISPATCH_DECLARE_CB_, __VA_ARGS__) \
^
/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu/npy_cpu_dispatch_config.h:94:10: note: expanded from macro 'NPY_MTARGETS_CONF_DISPATCH'
&&"Expected config header that generated by mod_features.multi_targets()";
^
../numpy/core/src/umath/_umath_tests.dispatch.c:20:1: error: expected identifier or '('
/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu/npy_cpu_dispatch_config.h:195:5: note: expanded from macro 'NPY_CPU_DISPATCH_DECLARE'
NPY_MTARGETS_CONF_BASELINE(NPY_CPU_DISPATCH_DECLARE_BASE_CB_, __VA_ARGS__)
^
/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu/npy_cpu_dispatch_config.h:92:10: note: expanded from macro 'NPY_MTARGETS_CONF_BASELINE'
&&"Expected config header that generated by mod_features.multi_targets()";
^
6 errors generated.
[45/306] Compiling C object numpy/core/lib_uma...seline.a.p/src_umath__umath_tests.dispatch.c.
FAILED: numpy/core/lib_umath_tests.dispatch.h_baseline.a.p/src_umath__umath_tests.dispatch.c.o
/usr/bin/clang -Inumpy/core/lib_umath_tests.dispatch.h_baseline.a.p -Inumpy/core -I../numpy/core -Inumpy/core/include -I../numpy/core/include -I../numpy/core/src/common -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.11/include/python3.11 -I/opt/homebrew/opt/llvm/include -I/Users/konstantin.ignatov/github.com/numpy/numpy/build/meson_cpu -fcolor-diagnostics -Wall -Winvalid-pch -std=c99 -O2 -g -fno-strict-aliasing -ftrapping-math -DNPY_HAVE_NEON_VFPV4 -DNPY_HAVE_NEON_FP16 -DNPY_HAVE_NEON -DNPY_HAVE_ASIMD -DNPY_MTARGETS_BASELINE -MD -MQ numpy/core/lib_umath_tests.dispatch.h_baseline.a.p/src_umath__umath_tests.dispatch.c.o -MF numpy/core/lib_umath_tests.dispatch.h_baseline.a.p/src_umath__umath_tests.dispatch.c.o.d -o numpy/core/lib_umath_tests.dispatch.h_baseline.a.p/src_umath__umath_tests.dispatch.c.o -c ../numpy/core/src/umath/_umath_tests.dispatch.cRuntime information:
>>> # python setup.py build_ext -i # kinda works
>>> import sys, numpy; print(numpy.__version__); print(sys.version)
2.0.0.dev0+git20230817.bdcd1c8
3.11.4 (main, Jun 20 2023, 17:23:00) [Clang 14.0.3 (clang-1403.0.22.14.1)]
>>> import numpy
>>> numpy.show_runtime()
[{'numpy_version': '2.0.0.dev0+git20230817.bdcd1c8',
'python': '3.11.4 (main, Jun 20 2023, 17:23:00) [Clang 14.0.3 '
'(clang-1403.0.22.14.1)]',
'uname': uname_result(system='Darwin', node='kpp-s1', release='22.6.0', version='Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000', machine='arm64')},
{'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
'found': ['ASIMDHP', 'ASIMDDP'],
'not_found': ['ASIMDFHM']}}]
Context for the issue:
I was working on numpy sprint at EuroSciPy and couldn't finish the setup.
/cc @seberg
Activity
seiko2plus commentedon Aug 18, 2023
Have you tried using
spin build --clean? It seems that the configuration file for the_umath_tests.dispatch.htarget is empty for some reason. Though I'm not certain why this occurred, could it be due to configuration process was interpreted during certain stage.Could you provide the Meson log? You can find it at
build/meson-logs/meson-log.txt. Sharing the generated config file atbuild/numpy/core/_umath_tests.dispatch.hwould also help me better understand the issue. Thanks!code-of-kpp commentedon Aug 20, 2023
Yes, sorry for not clarifying this. Absolutely fresh setup, starting with git/git clone has the same problem.
meson-log.txt
Sure, let me know if I missed something that could help
seiko2plus commentedon Aug 20, 2023
The configuration stage appears to be correct, and the auto-generated file is not empty and well-defined. However, an empty inclusion has somehow been included within the following code section:
numpy/numpy/core/src/umath/_umath_tests.dispatch.c
Lines 14 to 18 in f884cf6
To address this issue, it's suggested that you apply the following patch to explicitly define the generated config macros and rebuild the code to determine if the build process passes:
However, before applying the patch, it would be helpful to provide the complete build log to gain better insights into the issue. You can generate the build log using the following command:
Then please attach the
numpy_spin_build.txtfile to assist in diagnosing the potential problem. Based on my understanding, it's possible that an empty header of_umath_tests.dispatch.hdefined else where in the searched directories might be overriding the expectedbuild/numpy/core/_umath_tests.dispatch.hor maybe not empty header but likely the disutils configuration header been generated in place through the command:
so please make sure of cleaning the directory before using spin build through:
code-of-kpp commentedon Aug 20, 2023
Yes, this seems to be the issue.
git clean -dxdid the trick, it's working now.I was removing build directory instead, apparently this isn't enough.
code-of-kpp commentedon Aug 20, 2023
Just in case, these are the files that were removed by
git clean, probably spin's--cleanflag should remove them too.mattip commentedon Aug 20, 2023
@stefanv is there a way to pass a list of files to extend
spin clean?seiko2plus commentedon Aug 20, 2023
@code-of-kpp Thank you for your assistance in investigating this bug. I've updated the reproducer code within the pull request description and have also opened a pull request that fixes this issue (#24468).
@mattip, It seems the easiest fix is to ensures that distutils generates these headers in the build directory, even with the inplace option enabled
rgommers commentedon Aug 21, 2023
We don't have such a command, and I don't think we should add it. It's going to be inherently unreliable just like
setup.py cleanwas not reliable. It's simply not possible to know what to delete I think, unless it's a simple alias forgit clean -xdf. So better to continue to recommend to usegit clean -xdfdirectly.code-of-kpp commentedon Aug 21, 2023
Thank you all!
A couple of notes (and feel free to let me know, if they deserve an issue on their own):