BLD: undefined symbol: cblas_sgemm when building with meson with pip install

Steps to reproduce:

Use pyenv to install python 3.12.0b2.
Clone numpy v1.25.0rc1.
pip install -r requirements
Uncomment build-backend = "mesonpy".
pip install --no-build-isolation ..

The OS I use is Archlinux, and openblas and cblas are installed. In case it is relevant, openblas.pc is present in /usr/lib/pkgconfig/.

Error message:

Traceback (most recent call last):
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/overrides.py", line 8, in <module>
    from numpy.core._multiarray_umath import (
ImportError: /home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so: undefined symbol: cblas_sgemm

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/__init__.py", line 125, in <module>
    from numpy.__config__ import show as show_config
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/__config__.py", line 4, in <module>
    from numpy.core._multiarray_umath import (
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.12 from "/home/paro/.pyenv/versions/3.12.0b2/bin/python"
  * The NumPy version is: "1.25.0rc1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: /home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so: undefined symbol: cblas_sgemm


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/__init__.py", line 130, in <module>
    raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.

Additional information:

ldd shows that the compiled library is not linking against BLAS at all.

~ $ ldd /home/paro/.pyenv/versions/3.12.0b2/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffcdc3e1000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f0dfa084000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f0dfa05f000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f0df9e75000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007f0dfa59c000)

Hit the same in Gentoo and was trying to minimise it but came across this! Thank you!

Meson builds for NumPy are still a WIP: the OpenBLAS interface detection and SIMD support is still in flux. Here are a few hints:

You should see, very early in the meson build, something like

    The Meson build system
    Version: 1.1.1
    Source dir: /project
    Build dir: /project/.mesonpy-xfjik4zm/build
    Build type: native build
    Project name: NumPy
    Project version: 2.0.0.dev0
    C compiler for the host machine: cc (gcc 10.2.1 "cc (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)")
    C linker for the host machine: cc ld.bfd 2.35-5
    C++ compiler for the host machine: c++ (gcc 10.2.1 "c++ (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)")
    C++ linker for the host machine: c++ ld.bfd 2.35-5
    Cython compiler for the host machine: cython (cython 0.29.35)
    Host machine cpu family: x86_64
    Host machine cpu: x86_64
    Program python found: YES (/opt/python/cp310-cp310/bin/python)
    Found pkg-config: /usr/bin/pkg-config (0.27.1)
    Has header "Python.h" with dependency python-3.10: YES
    Compiler for C supports arguments -fno-strict-aliasing: YES
    Library m found: YES
    Found CMake: /usr/local/bin/cmake (3.26.3)
    WARNING: CMake Toolchain: Failed to determine CMake compilers state
    WARNING: CMake Toolchain: Failed to determine CMake compilers state
    Run-time dependency openblas found: NO (tried pkgconfig and cmake)
    WARNING: CMake Toolchain: Failed to determine CMake compilers state
    Run-time dependency openblas found: YES 0.3.23
    Dependency openblas found: YES 0.3.23 (cached)

Note the last two lines which show it is finding OpenBLAS via pkgconfig. If you see that, the build should link to OpenBLAS.

Additionally, there is a way to configure NumPy to use a build of OpenBLAS with 64-bit interfaces instead of 32-bit ones. This is called ILP64, and is currently expressed in a few environment variables and C macros (our convention is to add a 64_ suffix to all the exported symbols). It is unlikely that the system OpenBLAS is built with those symbols, so you should make sure you are not using -DBLAS_SYMBOL_SUFFIX=64_, -DHAVE_BLAS_ILP64, or NPY_USE_BLAS_ILP64="1"

I guess this is related to https://github.com/numpy/numpy/pull/23838/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711R150 then, which at least I was applying.

Thanks for the hint, I'll look into it more now.

EDIT: I found what I was doing wrong -- I was passing -Dblas=blas instead of -Dblas=cblas (need the generic name in Gentoo as we have a different BLAS selection method) and my environment also seemed to have something broken which I've now fixed. Thanks!

If you really want the 64-bit interfaces like in our wheels, you can get OpenBLAS from https://anaconda.org/multibuild-wheels-staging/openblas-libs/files, expand the tarball, and point PKG_CONFIG_PATH to the pkgconfig directory. Make sure it works by checking that pkg-config openblas --cflags returns the path to the include files

I am now using the branch in gh-23838. My meson build does show these two lines:

  Run-time dependency openblas found: YES
  Dependency openblas found: YES unknown (cached)

It seems my system openblas doesn't indicate its version in the pkg-config file.

I am building with just pip install ., without any special flags. However it still doesn't link to openblas. What can I do to fix this?

How are you checking for OpenBLAS? Could you be picking up a static library?

I have the pkg-config file /usr/lib/pkgconfig/openblas.pc:

libdir=/usr/lib
libsuffix=
includedir=/usr/include/openblas

openblas_config=USE_64BITINT= NO_CBLAS=1 NO_LAPACK=1 NO_LAPACKE=1 DYNAMIC_ARCH=ON DYNAMIC_OLDER=OFF NO_AFFINITY=ON USE_OPENMP=1 ZEN MAX_THREADS=64
Name: OpenBLAS
Description: OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version
Version:
URL: https://github.com/xianyi/OpenBLAS
Libs: -fopenmp -L${libdir} -lopenblas${libsuffix}
Cflags: -I${includedir}

(the version is indeed left empty, although the system package manager says it's 0.3.23)

And the dynamic libraries libopenblas.so, libopenblas.so.0, libopenblas.so.0.3 are in /usr/lib/.

In build/build.ninja, under

build numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so

I see this

 LINK_ARGS = -Wl,--as-needed -Wl,--allow-shlib-undefined -shared -fPIC -Wl,--start-group numpy/core/libnpymath.a /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libopenblas.so -Wl,--end-group

So... this means that I need to change the LINK_ARGS? But build.ninja is generated by meson. How do I change it?

My goal was to guide you to how you can debug the build. Do you see those link arguments? You can work backwards from there trying to figure out where blas/lapack detection is failing.

@mattip

In my build meson actually finds the libopenblas.so. This is what I found in build.ninja.

 LINK_ARGS = -Wl,--as-needed -Wl,--allow-shlib-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group numpy/core/libnpymath.a -fopenmp /usr/lib/libopenblas.so -Wl,--end-group

(By the way in my system (Archlinux) libcblas is installed separately and is a standalone dynamic library libcblas.so. The build doesn't seem to link to libcblas.so, while the prebuilt numpy for Python 3.11 from my system package manager links to it. Is this relevant?)

Update:

I realized that my system provides openblas as a drop-in replacement for netlib blas. So although I have openblas installed, maybe I actually need to build numpy to link to blas instead of openblas...?

Strangely, after I change openblas to blas in meson.build, the generated ninja file still uses libopenblas.so. The meson build message is now this:

  Run-time dependency blas found: YES
  Run-time dependency cblas found: YES 3.11.0
  Dependency blas found: YES unknown (cached)

But LINK_ARGS in build.ninja remains unchanged.

Update 2:

The content of blas.pc is the same as openblas.pc, so linking to openblas is expected. I still don't know why it doesn't link to openblas despite that LINK_ARGS seems correct. (is it? @mattip)

It is hard to debug this. You need to stare at the different options until they start to make sense. If the LINK_ARGS for numpy/core/_multiarray_umath.cpython-*.so look corrrect, then you should try looking at the compile args. Perhaps something is missing there.

So... it seems numpy does need to link to cblas. In my system, the symbol cblas_sgemm is exported by libcblas.so, not libopenblas.so. Although the meson build does define the cblas dependency variable, it doesn't actually use it in the module _multiarray_umath. I added cblas to the dependencies of _multiarray_umath and it worked.

Although the meson build does define the cblas dependency variable, it doesn't actually use it in the module _multiarray_umath. I added cblas to the dependencies of _multiarray_umath and it worked.

If this is something you think others will run into, it would be nice to summarize your changes and see how we can integrate them into the meson build.

I made two changes:

Use BLAS instead of OpenBLAS in meson.py. Maybe we should check if the user has cblas?

@@ -12,8 +12,8 @@ project(
     'b_ndebug=if-release',
     'c_std=c99',
     'cpp_std=c++17',
-    'blas=openblas',
-    'lapack=openblas',
+    'blas=blas',
+    'lapack=blas',
     'pkgconfig.relocatable=true',
   ],
 )

Add cblas to the dependencies of _multiarray_umath in numpy/core/meson.py. Because cblas is [] when there is no cblas, this will also work when there is no cblas.

@@ -859,7 +859,7 @@ py.extension_module('_multiarray_umath',
     'src/npymath',
     'src/umath',
   ],
-  dependencies: blas,
+  dependencies: [blas, cblas],
   link_with: npymath_lib,
   install: true,
   subdir: 'numpy/core',

Yes, the cblas support is fairly incomplete - I'll work on that in main soon. I'll also note that Arch Linux is the only distro where OpenBLAS is built without LAPACK support, and that's not yet handled automatically. So you'll need Netlib LAPACK as a dependency, and build with -Dlapack=lapack.

Add cblas to the dependencies of _multiarray_umath in numpy/core/meson.py. Because cblas is [] when there is no cblas, this will also work when there is no cblas.

I believe we unconditionally require CBLAS wherever we use BLAS, if BLAS is present and we're not using our own C implementation for matmul & co. So we may fix it more robustly by adding something like:

if have_blas
  blas_dep = declare_dependency(
    dependencies: [blas, cblas],
    compile_args: '-DHAVE_CBLAS',
  )
endif

and then everywhere else use dependencies: blas_dep.

So Archlinux just released a new version of OpenBLAS that includes cblas symbols and LAPACK support... Seems the modifications are no longer necessary.

However, if the user wants to use the package blas-openblas, the fore mentioned modifications are still useful.

@refparo

Ah, that is awesome, that solves a pain point - thanks for pointing that out @refparo!

BLD: improve handling of CBLAS, add `-Duse-ilp64` build option

@refparo

@refparo and @thesamesam this issue should be fixed by gh-23984. I tested on Arch Linux, not on Gentoo. There's more to do for the overall BLAS/LAPACK detection to make it >= numpy.distutils in functionality, but I hope this solves the immediate problem. If you could confirm, that would be much appreciated.

BLD: improve handling of CBLAS, add `-Duse-ilp64` build option (#23984)

Thanks, I've got this on my queue to look at now. Will comment again with results.

Navigation Menu

Repository navigation

BLD: undefined symbol: cblas_sgemm when building with meson with pip install #23909

Steps to reproduce:

Error message:

Additional information:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

BLD: undefined symbol: cblas_sgemm when building with meson with pip install #23909

Description

Steps to reproduce:

Error message:

Additional information:

Activity

thesamesam commented on Jun 12, 2023

mattip commented on Jun 12, 2023

thesamesam commented on Jun 12, 2023

mattip commented on Jun 12, 2023

refparo commented on Jun 12, 2023

mattip commented on Jun 12, 2023

refparo commented on Jun 12, 2023

mattip commented on Jun 12, 2023

refparo commented on Jun 12, 2023

mattip commented on Jun 12, 2023

refparo commented on Jun 12, 2023

mattip commented on Jun 12, 2023

refparo commented on Jun 13, 2023

mattip commented on Jun 13, 2023

refparo commented on Jun 13, 2023

rgommers commented on Jun 15, 2023

rgommers commented on Jun 15, 2023

refparo commented on Jun 15, 2023

rgommers commented on Jun 15, 2023

rgommers commented on Jun 19, 2023

thesamesam commented on Jun 27, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions