Repository navigation
BLD: undefined symbol: cblas_sgemm when building with meson with pip install #23909
Description
Activity
- changed the title
[-]undefined symbol: cblas_sgemm when building with meson with pip install[/-][+]BLD: undefined symbol: cblas_sgemm when building with meson with pip install[/+]on Jun 11, 2023 Hit the same in Gentoo and was trying to minimise it but came across this! Thank you!
Meson builds for NumPy are still a WIP: the OpenBLAS interface detection and SIMD support is still in flux. Here are a few hints:
You should see, very early in the meson build, something like
The Meson build system Version: 1.1.1 Source dir: /project Build dir: /project/.mesonpy-xfjik4zm/build Build type: native build Project name: NumPy Project version: 2.0.0.dev0 C compiler for the host machine: cc (gcc 10.2.1 "cc (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)") C linker for the host machine: cc ld.bfd 2.35-5 C++ compiler for the host machine: c++ (gcc 10.2.1 "c++ (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)") C++ linker for the host machine: c++ ld.bfd 2.35-5 Cython compiler for the host machine: cython (cython 0.29.35) Host machine cpu family: x86_64 Host machine cpu: x86_64 Program python found: YES (/opt/python/cp310-cp310/bin/python) Found pkg-config: /usr/bin/pkg-config (0.27.1) Has header "Python.h" with dependency python-3.10: YES Compiler for C supports arguments -fno-strict-aliasing: YES Library m found: YES Found CMake: /usr/local/bin/cmake (3.26.3) WARNING: CMake Toolchain: Failed to determine CMake compilers state WARNING: CMake Toolchain: Failed to determine CMake compilers state Run-time dependency openblas found: NO (tried pkgconfig and cmake) WARNING: CMake Toolchain: Failed to determine CMake compilers state Run-time dependency openblas found: YES 0.3.23 Dependency openblas found: YES 0.3.23 (cached)Note the last two lines which show it is finding OpenBLAS via pkgconfig. If you see that, the build should link to OpenBLAS.
Additionally, there is a way to configure NumPy to use a build of OpenBLAS with 64-bit interfaces instead of 32-bit ones. This is called ILP64, and is currently expressed in a few environment variables and C macros (our convention is to add a
64_suffix to all the exported symbols). It is unlikely that the system OpenBLAS is built with those symbols, so you should make sure you are not using-DBLAS_SYMBOL_SUFFIX=64_,-DHAVE_BLAS_ILP64, orNPY_USE_BLAS_ILP64="1"I guess this is related to https://github.com/numpy/numpy/pull/23838/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711R150 then, which at least I was applying.
Thanks for the hint, I'll look into it more now.
EDIT: I found what I was doing wrong -- I was passing
-Dblas=blasinstead of-Dblas=cblas(need the generic name in Gentoo as we have a different BLAS selection method) and my environment also seemed to have something broken which I've now fixed. Thanks!If you really want the 64-bit interfaces like in our wheels, you can get OpenBLAS from https://anaconda.org/multibuild-wheels-staging/openblas-libs/files, expand the tarball, and point PKG_CONFIG_PATH to the pkgconfig directory. Make sure it works by checking that
pkg-config openblas --cflagsreturns the path to the include filesI am now using the branch in gh-23838. My meson build does show these two lines:
Run-time dependency openblas found: YES Dependency openblas found: YES unknown (cached)It seems my system openblas doesn't indicate its version in the pkg-config file.
I am building with just
pip install ., without any special flags. However it still doesn't link to openblas. What can I do to fix this?How are you checking for OpenBLAS? Could you be picking up a static library?
I have the pkg-config file
/usr/lib/pkgconfig/openblas.pc:libdir=/usr/lib libsuffix= includedir=/usr/include/openblas openblas_config=USE_64BITINT= NO_CBLAS=1 NO_LAPACK=1 NO_LAPACKE=1 DYNAMIC_ARCH=ON DYNAMIC_OLDER=OFF NO_AFFINITY=ON USE_OPENMP=1 ZEN MAX_THREADS=64 Name: OpenBLAS Description: OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version Version: URL: https://github.com/xianyi/OpenBLAS Libs: -fopenmp -L${libdir} -lopenblas${libsuffix} Cflags: -I${includedir}(the version is indeed left empty, although the system package manager says it's 0.3.23)
And the dynamic libraries
libopenblas.so,libopenblas.so.0,libopenblas.so.0.3are in/usr/lib/.In
build/build.ninja, underbuild numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.soI see this
LINK_ARGS = -Wl,--as-needed -Wl,--allow-shlib-undefined -shared -fPIC -Wl,--start-group numpy/core/libnpymath.a /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libopenblas.so -Wl,--end-groupSo... this means that I need to change the
LINK_ARGS? Butbuild.ninjais generated by meson. How do I change it?My goal was to guide you to how you can debug the build. Do you see those link arguments? You can work backwards from there trying to figure out where
blas/lapackdetection is failing.In my build meson actually finds the
libopenblas.so. This is what I found inbuild.ninja.LINK_ARGS = -Wl,--as-needed -Wl,--allow-shlib-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group numpy/core/libnpymath.a -fopenmp /usr/lib/libopenblas.so -Wl,--end-group(By the way in my system (Archlinux) libcblas is installed separately and is a standalone dynamic library
libcblas.so. The build doesn't seem to link tolibcblas.so, while the prebuilt numpy for Python 3.11 from my system package manager links to it. Is this relevant?)
Update:
I realized that my system provides openblas as a drop-in replacement for netlib blas. So although I have openblas installed, maybe I actually need to build numpy to link to blas instead of openblas...?
Strangely, after I change openblas to blas in
meson.build, the generated ninja file still useslibopenblas.so. The meson build message is now this:Run-time dependency blas found: YES Run-time dependency cblas found: YES 3.11.0 Dependency blas found: YES unknown (cached)But
LINK_ARGSinbuild.ninjaremains unchanged.
Update 2:
The content of
blas.pcis the same asopenblas.pc, so linking to openblas is expected. I still don't know why it doesn't link to openblas despite thatLINK_ARGSseems correct. (is it? @mattip)It is hard to debug this. You need to stare at the different options until they start to make sense. If the
LINK_ARGSfornumpy/core/_multiarray_umath.cpython-*.solook corrrect, then you should try looking at the compile args. Perhaps something is missing there.So... it seems numpy does need to link to cblas. In my system, the symbol
cblas_sgemmis exported bylibcblas.so, notlibopenblas.so. Although the meson build does define thecblasdependency variable, it doesn't actually use it in the module_multiarray_umath. I addedcblasto the dependencies of_multiarray_umathand it worked.Although the meson build does define the
cblasdependency variable, it doesn't actually use it in the module_multiarray_umath. I addedcblasto the dependencies of_multiarray_umathand it worked.If this is something you think others will run into, it would be nice to summarize your changes and see how we can integrate them into the meson build.
I made two changes:
- Use BLAS instead of OpenBLAS in
meson.py. Maybe we should check if the user hascblas?
@@ -12,8 +12,8 @@ project( 'b_ndebug=if-release', 'c_std=c99', 'cpp_std=c++17', - 'blas=openblas', - 'lapack=openblas', + 'blas=blas', + 'lapack=blas', 'pkgconfig.relocatable=true', ], )
- Add
cblasto the dependencies of_multiarray_umathinnumpy/core/meson.py. Becausecblasis[]when there is nocblas, this will also work when there is nocblas.
@@ -859,7 +859,7 @@ py.extension_module('_multiarray_umath', 'src/npymath', 'src/umath', ], - dependencies: blas, + dependencies: [blas, cblas], link_with: npymath_lib, install: true, subdir: 'numpy/core',
- Use BLAS instead of OpenBLAS in
Yes, the
cblassupport is fairly incomplete - I'll work on that inmainsoon. I'll also note that Arch Linux is the only distro where OpenBLAS is built without LAPACK support, and that's not yet handled automatically. So you'll need Netlib LAPACK as a dependency, and build with-Dlapack=lapack.Add
cblasto the dependencies of_multiarray_umathinnumpy/core/meson.py. Becausecblasis[]when there is nocblas, this will also work when there is nocblas.I believe we unconditionally require CBLAS wherever we use BLAS, if BLAS is present and we're not using our own C implementation for matmul & co. So we may fix it more robustly by adding something like:
if have_blas blas_dep = declare_dependency( dependencies: [blas, cblas], compile_args: '-DHAVE_CBLAS', ) endif
and then everywhere else use
dependencies: blas_dep.So Archlinux just released a new version of OpenBLAS that includes cblas symbols and LAPACK support... Seems the modifications are no longer necessary.
However, if the user wants to use the package
blas-openblas, the fore mentioned modifications are still useful.- added 2 commits that reference this issue
on Jun 19, 2023 @refparo and @thesamesam this issue should be fixed by gh-23984. I tested on Arch Linux, not on Gentoo. There's more to do for the overall BLAS/LAPACK detection to make it
>= numpy.distutilsin functionality, but I hope this solves the immediate problem. If you could confirm, that would be much appreciated.- added a commit that references this issue
on Jun 26, 2023 Thanks, I've got this on my queue to look at now. Will comment again with results.
Reacted by Ralf Gommers
Steps to reproduce:
pip install -r requirementsbuild-backend = "mesonpy".pip install --no-build-isolation ..The OS I use is Archlinux, and
openblasandcblasare installed. In case it is relevant,openblas.pcis present in/usr/lib/pkgconfig/.Error message:
Additional information:
lddshows that the compiled library is not linking against BLAS at all.