Skip to content

BUG: meson detection of openblas builds an executable without using pkg-config --static --libs #24827

Closed
Closed
BUG: meson detection of openblas builds an executable without using pkg-config --static --libs#24827

Description

@mattip
mattip
opened on Sep 28, 2023

I see this in the logs when building:

snippet from the logfile 
Determining dependency 'openblas' with pkg-config executable '/usr/bin/pkg-config'
env[PKG_CONFIG_PATH]: /numpy/.openblas
-----------
Called: `/usr/bin/pkg-config --modversion openblas` -> 0
stdout:
0.3.23.dev
-----------
env[PKG_CONFIG_PATH]: /numpy/.openblas
-----------
Called: `/usr/bin/pkg-config --cflags openblas` -> 0
stdout:
-I/tmp/venv3/lib/python3.10/site-packages/scipy_openblas64/include -DBLAS_SYMBOL_SUFFIX=64_ -DHAVE_BLAS_ILP64
-----------
env[PKG_CONFIG_PATH]: /numpy/.openblas
env[PKG_CONFIG_ALLOW_SYSTEM_LIBS]: 1
-----------
Called: `/usr/bin/pkg-config --libs openblas` -> 0
env[PKG_CONFIG_PATH]: /numpy/.openblas
-----------
Called: `/usr/bin/pkg-config --libs openblas` -> 0
Run-time dependency openblas found: YES 0.3.23.dev
Running compile:
Working directory:  /numpy/build/meson-private/tmpy93gmgzu
Code:
 
      #ifndef BLAS_SYMBOL_SUFFIX
      #  define BLAS_SYMBOL_SUFFIX
      #endif
      #define EXPAND(suffix) cblas_ddot ## suffix
      #define DDOT(suffix) EXPAND(suffix)

      #include <cblas.h>

      int main(int argc, const char *argv[])
      {
          double a[4] = {1,2,3,4};
          double b[4] = {5,6,7,8};
          return DDOT(BLAS_SYMBOL_SUFFIX)(4, a, 1, b, 1) > 10;
      }
      
-----------
Command line: `cc -I/tmp/venv3/lib/python3.10/site-packages/scipy_openblas64/include /numpy/build/meson-private/tmpy93gmgzu/testfile.c -o /numpy/build/meson-private/tmpy93gmgzu/output.exe -DBLAS_SYMBOL_SUFFIX=64_ -DHAVE_BLAS_ILP64 -D_FILE_OFFSET_BITS=64 -O0 -std=c99` -> 1

My openblas.pc file is this

openblas.pc 
libdir=/tmp/venv3/lib/python3.10/site-packages/scipy_openblas64/lib
includedir=/tmp/venv3/lib/python3.10/site-packages/scipy_openblas64/include
openblas_config= OpenBLAS 0.3.23.dev  USE64BITINT DYNAMIC_ARCH NO_AFFINITY Zen MAX_THREADS=64
version=0.3.23.dev
extralib=-lm -lpthread -lgfortran -lm -lpthread -lgfortran
Name: openblas
Description: OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version
Version: ${version}
URL: https://github.com/xianyi/OpenBLAS
Libs: 
Libs.private: ${extralib} -L${libdir} -llibopenblas_python
Cflags: -I${includedir} -DBLAS_SYMBOL_SUFFIX=64_ -DBLAS_SYMBOL_SUFFIX=64_ -DHAVE_BLAS_ILP64

So when running non-static compilation, pkg-config openblas --libs will return an empty string (which is what I want, I will inject libopenblas_python into the executable's namespace by loading the DLL/SO before loading numpy), but when compiling the test code, meson should get the link arguments via pkg-config openblas --static --libs which will return the needed -L/tmp/venv3/lib/python3.10/site-packages/scipy_openblas64/lib -lm -lpthread -lgfortran -lm -lpthread -lgfortran -llibopenblas_python

The problematic compilation comes from numpy/meson.build here. How can that cc.links call be told "use static linking" ?

Activity

  1. mattip commented on Sep 28, 2023

    @mattip
    mattip
    on Sep 28, 2023
    MemberAuthor

    This diff solves the problem for the openblas-wheels #24749 build

    $ git diff numpy/meson.build
diff --git a/numpy/meson.build b/numpy/meson.build
index e2dc683bf..e9adfa2a4 100644
--- a/numpy/meson.build
+++ b/numpy/meson.build
@@ -98,6 +98,7 @@ if blas_name == 'openblas'
     _openblas_names = ['openblas', 'OpenBLAS']
   endif
   blas = dependency(_openblas_names, required: false)
+  blas_static = dependency(_openblas_names, required: false, static: true)
 elif blas_name.to_lower() == 'accelerate'
   # macOS 13.3+ has updated interfaces aligned with BLAS/LAPACK 3.9.1.  Use them if available.
   if macOS13_3_or_later
@@ -115,8 +116,10 @@ elif blas_name.to_lower() == 'accelerate'
     endif
     blas = dependency('Accelerate')
   endif
+  blas_static = blas
 else
   blas = dependency(blas_name, required: false)
+  blas_static = dependency(blas_name, required: false, static: true)
 endif
 have_blas = blas.found()
 cblas = []
@@ -160,7 +163,7 @@ if have_blas
           return DDOT(BLAS_SYMBOL_SUFFIX)(4, a, 1, b, 1) > 10;
       }
       ''',
-      dependencies: blas,
+      dependencies: blas_static,
       args: probe_args,
       name: 'CBLAS',
     )

  2. mattip commented on Sep 28, 2023

    @mattip
    mattip
    on Sep 28, 2023
    MemberAuthor

    Will wait a bit for comments and corrections, feel free to close if this looks OK

  3. rgommers commented on Sep 28, 2023

    @rgommers
    rgommers
    on Sep 28, 2023
    Member

    We never use static linking for BLAS/LAPACK, so this diff can't be right I think.

    The problematic compilation comes from numpy/meson.build here. How can that cc.links call be told "use static linking"

    If that's the actual problem, that cc.links call will disappear completely soon enough - we'll be asking Meson for "BLAS with CBLAS" and checks will happen under the hood.

  4. mattip commented on Sep 28, 2023

    @mattip
    mattip
    on Sep 28, 2023
    MemberAuthor

    I hope the meson checks under the hood know to differentiate between building an executable (like in the cc.links call) and the normal build of a shared library

  5. rgommers commented on Sep 28, 2023

    @rgommers
    rgommers
    on Sep 28, 2023
    Member

    I don't think this has anything to do with static linking. It's a normal check for whether a piece of code with a header links. The problem you're having is the use of dependencies: blas inside that cc.links call I think - that seems to do something unwanted for your use case.

  6. mattip commented on Sep 28, 2023

    @mattip
    mattip
    on Sep 28, 2023
    MemberAuthor

    It's a normal check for whether a piece of code with a header links.

    The cc.link call tries to make an executable, which will want to resolve all the symbols at link time, which means it must use the full set of link libraries. But using the scipy-openblas wheels does a trick: it does not link to any openblas library. Instead, it loads the symbols at runtime globally so the executable can resolve them. Note how the patch changes the dependencies: blas to dependencies: blas_static in order to work around this.

    Then the rest of the code continues as normal, throwing away the blas_static dependency and continuing with blas

  7. rgommers commented on Sep 28, 2023

    @rgommers
    rgommers
    on Sep 28, 2023
    Member

    So when running non-static compilation, pkg-config openblas --libs will return an empty string (which is what I want, I will inject libopenblas_python into the executable's namespace by loading the DLL/SO before loading numpy),

    This is going to cause Meson to be confused, so the simpler solution is to simply not try to detect blas/lapack and cblas. If we specify scipy-openblas*, we already know it exists and that it was built with CBLAS. No need to try to confirm that.

  8. mattip commented on Sep 29, 2023

    @mattip
    mattip
    on Sep 29, 2023
    MemberAuthor

    simpler solution is to simply not try to detect blas/lapack and cblas if we specify scipy-openblas

    That does sound simpler. Thanks.

  9. mattip
    closed this as completedon Sep 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions