Repository navigation
BUG: on OpenBSD/i386 numpy segfaults while running tests (1.20rc1 to 1.22.0 crash / 1.19.5 is the last known working version) #20799
Description
Describe the issue:
Starting with numpy 1.20rc1 and also happening in every release through 1.22.0, the tests cause a segfault on OpenBSD/i386. I've tested this with both python 3.8 and python 3.9. numpy 1.19.5 appears to allow the test suite to run.
The backtrace from gdb shows "solve_diophantine" may be where it's crashing?
In case helpful to know, on OpenBSD, netlib Blas is used.
Reproduce the code example:
$ egdb python3
GNU gdb (GDB) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "i386-unknown-openbsd7.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python3...
(No debugging symbols found in python3)
(gdb) run
Starting program: /usr/local/bin/python3
Python 3.8.12 (default, Jan 11 2022, 03:22:43)
[Clang 13.0.0 ] on openbsd7
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.test(verbose=2)
NumPy version 1.20.0rc1
NumPy relaxed strides checking option: True
NumPy CPU features: SSE SSE2 SSE3* SSSE3* SSE41* POPCNT* SSE42* AVX? F16C* FMA3? AVX2? AVX512F? AVX512CD? AVX512_KNL? AVX512_KNM? AVX512_SKX? AVX512_CLX? AVX512_CNL? AVX512_ICL?
===================================================================================================== test session starts ======================================================================================================
platform openbsd7 -- Python 3.8.12, pytest-6.2.5, py-1.8.2, pluggy-1.0.0
rootdir: /home/didickman/numpy, configfile: pytest.ini
plugins: hypothesis-6.35.0
collecting 11938 items / 4 errors / 11934 selected
Program received signal SIGSEGV, Segmentation fault.
0x0aba548b in solve_diophantine () from /usr/local/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38.so
(gdb) bt
#0 0x0aba548b in solve_diophantine () from /usr/local/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38.so
#1 0x0aba681b in solve_may_share_memory () from /usr/local/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38.so
#2 0x0ab7c8bf in PyUFunc_GenericFunction_int () from /usr/local/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38.so
#3 0x0ab7d001 in ufunc_generic_call () from /usr/local/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38.so
#4 0x08c40e6f in _PyObject_MakeTpCall () from /usr/local/lib/libpython3.8.so.0.0
#5 0x08c42e04 in object_vacall () from /usr/local/lib/libpython3.8.so.0.0
#6 0x08c42f71 in PyObject_CallFunctionObjArgs () from /usr/local/lib/libpython3.8.so.0.0
#7 0x0aa02e31 in PyArray_GenericInplaceBinaryFunction () from /usr/local/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38.so
#8 0x08c2a072 in PyNumber_InPlaceMultiply () from /usr/local/lib/libpython3.8.so.0.0
#9 0x08cffd87 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#10 0x08d0875e in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.8.so.0.0
#11 0x08c4193a in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#12 0x08c411be in PyVectorcall_Call () from /usr/local/lib/libpython3.8.so.0.0
#13 0x08c412ac in PyObject_Call () from /usr/local/lib/libpython3.8.so.0.0
#14 0x0a93935c in array_implement_array_function () from /usr/local/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38.so
#15 0x08c41481 in cfunction_call_varargs () from /usr/local/lib/libpython3.8.so.0.0
#16 0x08c41f8a in PyCFunction_Call () from /usr/local/lib/libpython3.8.so.0.0
#17 0x08c40e6f in _PyObject_MakeTpCall () from /usr/local/lib/libpython3.8.so.0.0
#18 0x08d07b86 in call_function () from /usr/local/lib/libpython3.8.so.0.0
#19 0x08d04965 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#20 0x08d0875e in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.8.so.0.0
#21 0x08c4193a in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#22 0x08d07b44 in call_function () from /usr/local/lib/libpython3.8.so.0.0
<snipped by @seberg>Error message:
Program received signal SIGSEGV, Segmentation fault.NumPy/Python version information:
1.20.0rc1 3.8.12 (default, Jan 11 2022, 03:22:43)
[Clang 13.0.0 ]
Activity
seberg commentedon Jan 12, 2022
@didickman just to get everyone on the same page, can you say how NumPy was installed, and if it was a wheel, maybe even see if compiling it locally helps?
It seems like a failure in
solve_diophantine, although the most interesting thing that does is passing around some 128bit integers (which need to be packed into a struct). It might be that the function doing that must be inlined for ABI reasons, but fail to be inlined or so. (Just a guess, I know such issues are possible, but I am not sure if it makes at all sense here.)didickman commentedon Jan 12, 2022
For the OpenBSD port we download the source from pypi and compile it ourselves. For example for 1.20rc1 I downloaded https://pypi.io/packages/source/n/numpy/numpy-1.20.0rc1.zip. If there's any test code I could try to run here that might let me debug the issue I could try that. I just don't know my way around the test suite well enough to know what to try look at/run.
seberg commentedon Jan 12, 2022
No, running the test suite is completely fine and good enough. My random idea was basically replacing all
static NPY_INLINEwithNPY_FINLINEinnumpy/core/src/common/npy_extint128.hto see if that might help.charris commentedon Jan 12, 2022
Out of curiosity, what compiler are you using. IIRC, the memory overlap detection uses 128 bit integers in parts, which might be a problem.
charris commentedon Jan 12, 2022
Do things work on x86_64?
didickman commentedon Jan 12, 2022
The base compiler on OpenBSD/i386 is clang 13:
$ cc --version
OpenBSD clang version 13.0.0
Target: i386-unknown-openbsd7.0
Thread model: posix
InstalledDir: /usr/bin
Will have to come back on how things look on x86_64. Thanks!
seberg commentedon Jan 12, 2022
If you end up compiling again, maybe try including debug symbols and run gdb again, just to make sure we got the right idea? Not sure it will give us anything, but can't hurt?
However, I agree with Chuck that the only likely problem is the issue of passing a 128bit struct by value. Force-inlining the functions in the header I pointed out _should solve that issue, so it would be a good first try I think.
didickman commentedon Jan 17, 2022
unfortunately no change in behaviour when replacing
static NPY_INLINEwithNPY_FINLINEinnumpy/core/src/common/npy_extint128.hseberg commentedon Jan 17, 2022
@didickman it seems hard to make progress on the issue. Any chance you can get the debug symbols in, so that we at least know where exactly it crashes (also we can see if the
NPY_FINLINEdid what I hoped it would do)? If you are stuck on getting that, worst case, clone the current git (recursive for the submodules), and then try:(maybe compile with gdb, so could compile once first without to not get that. But should just work, hopefully.)
For example Debian ships i386, so I somewhat expect it is something to do with clang 13 + i386, which makes it rather specific...
didickman commentedon Jan 17, 2022
ok, i think i was able to get the debug symbols going (apologies, took me a bit to work it out with our ports infrastructure).
See some details below. I hope this is helpful to pinpoint what's going on.
sthen commentedon Aug 30, 2022
I see the same segfault with numpy 1.23.2 on OpenBSD/i386, including with Python 3.10. (I'm trying to prepare the OpenBSD ports tree to move to Python 3.10 by default, and numpy is the biggest sticking point). Slightly longer backtrace here so I'll attach it rather than in-line (some of the _PyEval_EvalFrame lines are pretty wide). I'm no expert at debugging Python code but if you have anything you'd like me to try I'll see if I can help.
numpy.bt.openbsd-i386.txt
And here's the log from building numpy in case there's anything helpful there.
py-numpy,python3.log
sthen commentedon Aug 30, 2022
Similar segv/backtrace if I build with GCC instead of Clang.
sthen commentedon Aug 30, 2022
Aha, after some bisecting it seems it was introduced in 8245b39 (the big "ENH: enable multi-platform SIMD compiler optimizations" diff). Moving back to 1.23.2 I've tested with "--cpu-dispatch min", which still segfaults, and "--disable-optimization" which is ok.
btw, I noticed 'numpy/core/src/multiarray/common.h:181: warning: "_ALIGN" redefined' in build log. They're unrelated (occur on all OpenBSD archs, are nothing new, and still occur if I rename numpy's _ALIGN macro).
13 remaining items
sthen commentedon Jan 9, 2023
charris commentedon Aug 1, 2024
Pushing this off again. @didickman @sthen Any news of progress?
didickman commentedon Aug 2, 2024
we’re now on version 1.25.2 and it was likely still a problem for this version since it looks like Stuart kept optimizations disabled for that update. Is there a newer version that’s expected to fix this? I have a local diff for 1.26.4 (although not 2.x yet). Do we need to get 2.x working on our side to test the fix? Separately, @sthen has the i386 ports machine. If needed I can set up a 32 bit box for testing on my end but it will take me a few weeks to pull the box from storage.
charris commentedon Aug 2, 2024
Not that I know of, but sometimes things get fixed by accident :) Trying with a later version might get more attention, though.
seberg commentedon Aug 2, 2024
Still as puzzling as always... Seems to be happening very early on? Maybe worth a shot to run it with
valgrind(yes, that is infinitely slow)?Of all the errors, I really can't see how a segfault can come about here, unless the compiler is doing some incorrect optimizations!?
sthen commentedon Aug 28, 2024
Same issue in 1.26.4
mattip commentedon Aug 29, 2024
On this niche platform, I think it is an OK solution to add the
--disable-optimizationflag. It is unlikely we are going to get very far debugging this: we would need to set up exact emulation of this 32-bit processor and OpenBSD.