BUG: mkl errors and segfaults with numpy 1.26.0 on i9-10900K #24846

Activity

1. sn6526

   added

   00 - Bug

   on Oct 3, 2023
2. kernel-loophole commented on Oct 3, 2023

   

   kernel-loophole

   on Oct 3, 2023

   Last edited by rgommers

   More actions

   HI @stuart-nolan after Running the same code in colab i got the following output.

   Details

   openblas64__info:
       libraries = ['openblas64_', 'openblas64_']
       library_dirs = ['/usr/local/lib']
       language = c
       define_macros = [('HAVE_CBLAS', None), ('BLAS_SYMBOL_SUFFIX', '64_'), ('HAVE_BLAS_ILP64', None)]
       runtime_library_dirs = ['/usr/local/lib']
   blas_ilp64_opt_info:
       libraries = ['openblas64_', 'openblas64_']
       library_dirs = ['/usr/local/lib']
       language = c
       define_macros = [('HAVE_CBLAS', None), ('BLAS_SYMBOL_SUFFIX', '64_'), ('HAVE_BLAS_ILP64', None)]
       runtime_library_dirs = ['/usr/local/lib']
   openblas64__lapack_info:
       libraries = ['openblas64_', 'openblas64_']
       library_dirs = ['/usr/local/lib']
       language = c
       define_macros = [('HAVE_CBLAS', None), ('BLAS_SYMBOL_SUFFIX', '64_'), ('HAVE_BLAS_ILP64', None), ('HAVE_LAPACKE', None)]
       runtime_library_dirs = ['/usr/local/lib']
   lapack_ilp64_opt_info:
       libraries = ['openblas64_', 'openblas64_']
       library_dirs = ['/usr/local/lib']
       language = c
       define_macros = [('HAVE_CBLAS', None), ('BLAS_SYMBOL_SUFFIX', '64_'), ('HAVE_BLAS_ILP64', None), ('HAVE_LAPACKE', None)]
       runtime_library_dirs = ['/usr/local/lib']
   Supported SIMD extensions in this NumPy install:
       baseline = SSE,SSE2,SSE3
       found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2
       not found = AVX512F,AVX512CD,AVX512_KNL,AVX512_KNM,AVX512_SKX,AVX512_CLX,AVX512_CNL,AVX512_ICL
   python compile flags
   -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g       -fstack-protector-strong -Wformat -Werror=format-security  -g -fwrapv -O2   
   mkl: True
   physical cores: 2
   logical cores: 2
   cpu min freq: 0.0
   cpu current freq: 2200.15
   cpu max freq: 0.0
   load average: 1.0537109375 0.6015625 0.23486328125
   Total memory: 13.61 GB
   Available memory: 12.64 GB
   Dotted two 4096x4096 matrices in 4.40 s.
   Dotted two vectors of length 524288 in 1.39 ms.
   SVD of a 2048x1024 matrix in 1.56 s.
   Cholesky decomposition of a 2048x2048 matrix in 0.31 s.
   

   Also if you wanted to run code here is link for colab notebook https://colab.research.google.com/drive/1hOXH9qPc0MU4RseMYJVCbmhuGTljus8q?usp=sharing
3. sn6526 commented on Oct 3, 2023

   

   sn6526

   on Oct 3, 2023

   Author

   More actions

   @kernel-loophole
   I'm not sure how your post is relevant to this bug report. Your output shows numpy using openblas, and it does not show the output detecting the cpu. If your suggesting using openblas, thank you - I am aware of that option.
4. seberg commented on Oct 3, 2023

   

   seberg

   on Oct 3, 2023

   Member

   More actions

   You should tell us how exactly you installed numpy and MKL, since we do not distribute NumPy linked against MKL.

   One possibility is that MKL might offer a 64bit and 32bit lapack and if you dynamically swap things out, weird things may happen (you seem to use a 64bit one, but not sure about symbol suffix). If we had a real issue in NumPy, I would expect more reports about it, so I think that is more likely and you need to clarify your setup. You can also try using: np.show_runtime() with threadpoolctl installed.
5. rgommers commented on Oct 3, 2023

   

   rgommers

   on Oct 3, 2023

   Member

   More actions

   You should tell us how exactly you installed numpy and MKL

   That would indeed be helpful. Up to 1.25.2, we linked MKL through the Single Dynamic Library (i.e., with -lmk_rt for libmkl_rt.so) in numpy.distutils. With 1.26.0, you're likely building with Meson, so both BLAS linking and SIMD flags had major changes.
6. sn6526 commented on Oct 3, 2023

   

   sn6526

   on Oct 3, 2023

   Last edited by sn6526

   Author

   More actions

   MKL is installed via:
   sudo apt install intel-oneapi-mkl intel-oneapi-mkl-devel

   numpy install is via a custom bash script. I'll indicate below what is different between numpy 1.25.2 and 1.26.0.

   the bash shell environment is set up with

          intel_setvars_sh="/opt/intel/oneapi/setvars.sh"                           
          if [ -f ${intel_setvars_sh} ]; then                                       
             source ${intel_setvars_sh} >/dev/null                                  
             export MKL_THREADING_LAYER=GNU                                         
             export MKL_INTERFACE_LAYER=GNU                                         
          fi                                                                        
   

   excerpts from the "custom bash script"

        git clone https://github.com/numpy/numpy.git numpy --recursive         
        cd numpy
        latestTag=$(git describe --tags $(git rev-list --tags --max-count=1))       
        git checkout ${latestTag}                                              
        export CFLAGS="-fopenmp -m64 -march=native -DNDEBUG -O3 -Wl,--no-as-needed" 
        export CXXFLAGS="-fopenmp -m64 -march=native -DNDEBUG -O3 -Wl,--no-as-needed"
        export LDFLAGS="-ldl -lm"
        export FFLAGS="-fopenmp -m64 -march=native -DNDEBUG -O3"                    
   

   For numpy 1.26.0 (on both the i5-4300U and the i9-10900K):

        python -m build -Csetup-args=-Dblas=MKL -Csetup-args=-Dlapack=MKL      
        pip install dist/numpy*.whl                                            
   

   For numpy 1.25.2 (on both the i5-4300U and the i9-10900K):

        ncp1=$(expr $(cat /proc/cpuinfo | grep processor | wc -l) + 1)          
        NPY_BLAS_ORDER='MKL' NPY_LAPACK_ORDER='MKL' python setup.py build -j ${ncp1} install                                                                      
   

   I'll check if mkl is using 64bit and 32bit lapack, if it is different between the i5-4300U and the i9-10900K, and report back in a bit. The mkl install does not change on the i9-10900K between the functional numpy 1.25.2 and non- functional numpy 1.26.0.
7. rgommers commented on Oct 3, 2023

   

   rgommers

   on Oct 3, 2023

   Member

   More actions

   • I imagine that the problem remains if you leave out the -march=native flag?
   • If you want to compare apples to apples, I think you want to install pkg-config and use -Dblas=mkl-sdl and -Dlapack=mkl-sdl (I don't know what CMake defaulted to in your current build)

   👍React with 👍1Reacted by sn6526
8. sn6526 commented on Oct 3, 2023

   

   sn6526

   on Oct 3, 2023

   Last edited by sn6526

   Author

   More actions

   • leave out the -march=native flag
   • install pkg-config and use -Dblas=mkl-sdl and -Dlapack=mkl-sdl

   give me a bit. I do need to check without -march=native - a hold over from my gentoo days and I'm not sure it does anything for me now.

   pkg-config should be installed. I'll try mkl-sdl
9. sn6526 commented on Oct 3, 2023

   

   sn6526

   on Oct 3, 2023

   Last edited by sn6526

   Author

   More actions

   Using
   python -m build -Csetup-args=-Dblas=mkl-sdl -Csetup-args=-Dlapack=mkl-sdl
   results in a functional numpy 1.26.0 on the i9-10900K.

   FWIW, removing -march=native (as well as -Wl,--no-as-needed) has no effect.

   I have seen your comment in #24808

   a plain pip install numpy with only MKL installed in a standard system location ... should just work

   and would like to use something like that. Past experience is that having 3 blas libraries installed (no amount of update-alternatives configuring seems to stop numpy from using openblas if it finds it) and wanting to tune cflags seems to get in the way. I hope this changes with meson.

   EDIT: the i5-4300U device is indeed using mkl_rt with numpy 1.26.0 despite the same (non mkl-sdl) build arguments. Scipy also now uses meson (I adapted my original numpy/meson/mkl based build commands from that), and on the i5-4300U does not default to mkl_rt (as it did before meson). I had not noticed the difference nor have I observed issues with scipy yet.

   The short of it is, this is beyond my ability to troubleshoot or fix. If there is something you would like me to test please let me know.

   Thank you for your fast and helpful responses.
10. rgommers commented on Oct 3, 2023

    

    rgommers

    on Oct 3, 2023

    Member

    More actions

    and would like to use something like that.

    Great - should arrive soon (working on it right now), I hope days rather than weeks.

    Thanks for digging into this @stuart-nolan. It sounds like you're good for now - I'll finish my updates to this detection machinery first, and will then circle back to this.

    I think for the default -Dblas=mkl we can aim for mkl_rt, but in addition I'd like to support ILP64 builds and threading options - for those, we do need the split libraries.

    👍React with 👍1Reacted by sn6526
11. sn6526 commented on Oct 3, 2023

    

    sn6526

    on Oct 3, 2023

    Author

    More actions

    @seberg

    One possibility is that MKL might offer a 64bit and 32bit lapack

    I'd like to be able to show what MKL is doing at run time. Perhaps there are clues in the MKL_VERBOSE output; however I don't have the knowledge at this time to decode that.

    For now, if I ldd the *.so files in the scipy package dir, I see the following (currently its the same on both devices)

    ldd _fblas.cpython-311-x86_64-linux-gnu.so
    libiomp5.so => /opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin/libiomp5.so (0x00007fc9f8600000)
    libmkl_core.so.2 => /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so.2 (0x00007fc9f4200000)
    libmkl_gf_ilp64.so.2 => /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_gf_ilp64.so.2 (0x00007fc9f3200000)
    libmkl_intel_thread.so.2 => /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_thread.so.2 (0x00007fc9efa00000)
    

    So all links go back to the intel64 directory (I do not even have a intel32 dir). Also it looks like mkl wants to use 64 bit integers (ilp64). IIRC, I saw similar with numpy 1.26.0 on the i9-10900K before reverting to mkl_rt which still points back to the intel64 dir. I do recall having issues trying to use 64 bit integers with mkl and another library (sundials?) but that is out of scope for this bug report.

    Thanks for the tip on threadpoolctl and for responding.
12. rgommers commented on Oct 10, 2023

    

    rgommers

    on Oct 10, 2023

    Member

    More actions

    This should be fixed now with gh-24893. It also adds a CI job with MKL that passed with layered LP64, layered ILP64 and LP64 SDL.
13. charris

    closed this as completedin #24893on Oct 10, 2023
14. sn6526 commented on Oct 15, 2023

    

    sn6526

    on Oct 15, 2023

    Last edited by sn6526

    Author

    More actions

    @rgommers

    First, ty for the above.

    If I build numpy 1.26.1 with -Csetup-args=-Dblas=mkl-sdl -Csetup-args=-Dlapack=mkl-sdl, I'm still good (i.e. I do not observe the segfault below).

    However, if I build numpy 1.26.1 with

    python -m build -Csetup-args=-Dblas=MKL -Csetup-args=-Dlapack=MKL -Csetup-args=-Duse-ilp64=true
    

    (64 bit integers do not seem to be the issue here. I rebuilt numpy w/ and wo/ -Duse-ilp64=true. Scipy 1.11.3 built with -Csetup-args=-Dblas=MKL -Csetup-args=-Dlapack=MKL is apparently built with 64 bit integer support by default - and, yes, I've tried rebuilding scipy after each build/install of numpy 1.26.1)

    I get a (new) segfualt from ipython after:

    ... <python code after # Cholesky Decomposition mentioned above - which now does not segfault>
    import scipy
    import scipy.sparse as sps
    matrix = sps.rand(1000, 1000, density=0.001) + sps.diags(np.random.rand(1000))   
    scipy.linalg.qr(matrix.toarray(),pivoting=True)
    
    Segmentation fault (core dumped)
    

    I'm not really sure where to go from here. Please advise.
15. rgommers commented on Oct 15, 2023

    

    rgommers

    on Oct 15, 2023

    Member

    More actions

    Thanks for testing @stuart-nolan.

    Scipy 1.11.3 built with -Csetup-args=-Dblas=MKL -Csetup-args=-Dlapack=MKL is apparently built with 64 bit integer support by default

    That can't be (at least without further manual twekas), since support for that current isn't implemented in SciPy's meson.build files.

    The use of =MKL (upper-case) means the detection is going through CMake, and I don't know off the top of my head what that defaults to - I'll check. It can be seen from the link arguments for MKL in build/build.ninja.

    I'm not really sure where to go from here.

    I expect that the problem is linking MKL in two different ways in numpy and scipy, and then MKL being unhappy after both packages get imported.

    Can you add how you installed MKL? It matters, because various MKL distributions are broken in different ways. I tried to exclude picking those up in the NumPy build, but going via CMake bypasses that logic. If you use lower-case =mkl, that may help.
16. sn6526 commented on Oct 15, 2023

    

    sn6526

    on Oct 15, 2023

    Last edited by sn6526

    Author

    More actions

    use lower-case =mkl

    After building numpy with

    python -m build -Csetup-args=-Dblas=mkl -Csetup-args=-Dlapack=mkl
    

    Note scipy built with upper case MKL

    python -m build -Csetup-args=-Dblas=MKL -Csetup-args=-Dlapack=MKL
    

    using a lower case mkl with scipy gives the build time error:

    ../scipy/meson.build:161:9: ERROR: Dependency "mkl" not found, tried pkgconfig and cmake
    

    I still see the same seg fault with scipy's qr (numpy's qr works as does sparseqr).

    I cannot rule out user (aka my) ignorance about properly configuring numpy/scipy/MKL with the latest build system changes. As I have multiple options to build and install functional numpy/scipy/MKL, I don't want to create too much noise here.

    That said, I am willing to provide logs or additional troubleshooting if it helps. E.g.:

    Warnings observed during building scipy

    [292/1628] Generating scipy/linalg/fblas_module with a custom command
    [293/1628] Compiling C object scipy/linalg/fblas.cpython-311-x86_64-linux-gnu.so.p/meson-generated..__fblasmodule.c.o
    scipy/linalg/_fblasmodule.c: In function ‘complex_double_from_pyobj’:
    scipy/linalg/_fblasmodule.c:221:47: warning: passing argument 1 of ‘PyArray_DATA’ from incompatible pointer type [-Wincompatible-pointer-types]
    221 | (*v).r = ((npy_cdouble *)PyArray_DATA(arr))->real;
    | ^~~
    | |
    | PyObject * {aka struct _object *}
    In file included from ../../../build-env-qszcrox7/lib/python3.11/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
    from ../../../build-env-qszcrox7/lib/python3.11/site-packages/numpy/core/include/numpy/arrayobject.h:5,
    from ../../../build-env-qszcrox7/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.h:13,
    from scipy/linalg/_fblasmodule.c:23:
    ../../../build-env-qszcrox7/lib/python3.11/site-packages/numpy/core/include/numpy/ndarraytypes.h:1532:29: note: expected ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject *’} but argument is of type ‘PyObject *’ {aka ‘struct _object *’}
    1532 | PyArray_DATA(PyArrayObject *arr)
    | ~~~~~~~~~~~~~~~^~~
    scipy/linalg/_fblasmodule.c:222:47: warning: passing argument 1 of ‘PyArray_DATA’ from incompatible pointer type [-Wincompatible-pointer-types]
    222 | (*v).i = ((npy_cdouble *)PyArray_DATA(arr))->imag;
    | ^~~
    | |
    | PyObject * {aka struct _object *}
    In file included from ../../../build-env-qszcrox7/lib/python3.11/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
    from ../../../build-env-qszcrox7/lib/python3.11/site-packages/numpy/core/include/numpy/arrayobject.h:5,
    from ../../../build-env-qszcrox7/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.h:13,
    from scipy/linalg/_fblasmodule.c:23:
    ../../../build-env-qszcrox7/lib/python3.11/site-packages/numpy/core/include/numpy/ndarraytypes.h:1532:29: note: expected ‘PyArrayObject *’ {aka ‘struct tagPyArry/ndarraytypes.h:1532:29: note: expected ‘PyArrayObject *’ {aka ‘struct tagPyArrayObject *’} but argument is of type ‘PyObject *’ {aka ‘struct _object *’}
    1532 | PyArray_DATA(PyArrayObject *arr)

    Can you add how you installed MKL? It matters

    Same as posted above:

    MKL install on ubuntu 22.04 lts

    MKL is installed via: sudo apt install intel-oneapi-mkl intel-oneapi-mkl-devel

    ...
    the bash shell environment is set up with

           intel_setvars_sh="/opt/intel/oneapi/setvars.sh"                           
           if [ -f ${intel_setvars_sh} ]; then                                       
              source ${intel_setvars_sh} >/dev/null                                  
              export MKL_THREADING_LAYER=GNU                                         
              export MKL_INTERFACE_LAYER=GNU                                         
           fi                                                                        
    

    If the above responses are missing something you would like to see wrt the MKL install, please let me know.

    FWIW, the bash environment is set both for building numpy/scipy and run time/testing. I've tried unsetting the MKL_THREADING_LAYER and MKL_INTERFACE_LAYER variables at run time and I still observe the segfault. I do need to review why I set these - I seem to recall it's from past troubleshooting building scipy with MKL but I'm not sure about that.

    That can't be (at least without further manual twekas), since support for that current isn't implemented in SciPy's meson.build files.

    My comment about scipy "defaulting to using MKL 64 bit integers" is based on

    cd ~/.pyenv/versions/p3114.mkl/lib/python3.11/site-packages/scipy/linalg/
    ldd _fblas.cpython-311-x86_64-linux-gnu.so _flapack.cpython-311-x86_64-linux-gnu.so | grep ilp64
            libmkl_gf_ilp64.so.2 => /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_gf_ilp64.so.2 (0x00007fccf5600000)
            libmkl_gf_ilp64.so.2 => /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_gf_ilp64.so.2 (0x00007faa4e000000)
    

    I'll defer to your greater knowledge and experience here and I'm also happy to grep through build/build.ninja if it comes to that.
17. rgommers commented on Oct 15, 2023

    

    rgommers

    on Oct 15, 2023

    Member

    More actions

    MKL is installed via: sudo apt install intel-oneapi-mkl intel-oneapi-mkl-devel

    That isn't part of the default Ubuntu package repository, so I think that's coming from https://apt.repos.intel.com/oneapi (as documented in this Intel install guide). That should be fine - and it's up-to-date (2023.2.0).

    My comment about scipy "defaulting to using MKL 64 bit integers" is based on

    Thanks. I'm surprised that that worked - guess there must be 32-bit symbols in that library too somehow.

    Warnings observed during building scipy

    Those warnings don't look related.

    As I have multiple options to build and install functional numpy/scipy/MKL, I don't want to create too much noise here.

    No worries at all - this is very useful feedback. Given how many ways there are to install MKL and that they're semi-broken in various ways, I'd like to make this as robust as possible.

    This is all still in motion, so I understand it's hard to figure out what options to pass to select MKL. The goal here is ensuring that NumPy and SciPy are built with the exact same MKL config. There should be two ways right now:

    1. If you build numpy with -Dblas=mkl (and same for lapack) or you don't provide any -D flag and rely on auto-detection, then you want to build SciPy with -Dblas=mkl-dynamic-lp64-seq
    2. Or, build both numpy and scipy with =mkl-sdl

    👍React with 👍1Reacted by sn6526
18. sn6526 commented on Oct 16, 2023

    

    sn6526

    on Oct 16, 2023

    Last edited by sn6526

    Author

    More actions

    @rgommers

    I suspect the scipy.linalg.qr segfault is due to MKL threading "interface". Using the hint you provided above for scipy/numpy/MKL build commands and inspecting meson_options.txt in the numpy build directory, I used:

    numpy 1.26.1

    python -m build -Csetup-args=-Dblas=mkl -Csetup-args=-Dlapack=mkl -Csetup-args=-Dmkl-threading=gomp
    

    scipy 1.11.3

    python -m build -Csetup-args=-Dblas=mkl-dynamic-lp64-gomp -Csetup-args=-Dlapack=mkl-dynamic-lp64-gomp
    

    scipy.linalg.qr no longer segfaults.

    Before, my builds above for both numpy and scipy were using iomp. For reasons I do not want to discus, I don't want intel threading (or 64 bit integers) but I tolerated them temporarily while working through this thinking it should not matter at this point. Unfortunately, threading matters.

    More importantly, I most definitely want to know how to configure numpy/scipy/MKL for threading and 32/64 bit integers should my requirements change. I understand that this is a work in progress. I hope the final result is a consistent "MKL configuration interface" between numpy and scipy.

    Thank you again for your help and the effort you are putting into this.

    EDIT: should anyone else find their way here, the build command

    python -m build -Csetup-args=-Dblas=mkl-dynamic-lp64-gomp -Csetup-args=-Dlapack=mkl-dynamic-lp64-gomp
    

    currently works for both scipy and numpy. I prefer using mkl-dynamic-lp64-gomp since it explicitly states the entire intent in a compact way (and it's what numpy.show_config() outputs).

    Lastly, it is possible the scipy.linalg.qr segfault I experienced is due to "user error." When building with iomp, I likely still exported -fopenmp in the cflags - I'm not sure if this is a compatible combination.