Repository navigation
numpy umath.cpython SIGILL (1.14.2) #10787
Description
Downgrading numpy to 1.11.3 helped
I'm getting a segfault when using numpy 1.14.2. Here's the gdb trace:
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `python tests/test_db_bq_loader.py'.
Program terminated with signal SIGILL, Illegal instruction.
#0 0x00007fb03004e400 in LONG_add_avx2 (args=<optimized out>,
dimensions=<optimized out>, steps=<optimized out>,
__NPY_UNUSED_TAGGEDfunc=<optimized out>)
from /opt/conda/envs/xx/lib/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so
(gdb)
Here is the code that can replicate this:
import numpy as np
import pandas as pd
from numpy.random import randint
data = [{'ds': '2017-01-01', 'a': 10, 'b': 20, 'c': 30}, {'ds': '2017-01-01', 'a': 11, 'b': 21, 'c': 31},{'ds': '2017-01-01', 'a': 12, 'b': 22, 'c': 32},{'ds': '2017-01-02', 'a': 13, 'b': 23, 'c': 33},{'ds': '2017-01-02', 'a': 14, 'b': 24, 'c': 34},{'ds': '2017-01-03', 'a': 15, 'b': 25, 'c': 35},{'ds': '2017-01-04', 'a': randint(1, 100), 'b': randint(1, 100), 'c': randint(1, 100)}]
df = pd.DataFrame(data)
np.array_equal(df.sort_values(['ds', 'a', 'b', 'c']).values, df.sort_values(['ds', 'a', 'b', 'c']).values)
- It's running in a docker container. proc/cpuinfo says GenuineIntel Xeon E5-2660
Python version and pip freeze
root@31dedfee1f82:/app/xxx# python --version
Python 3.6.4
root@31dedfee1f82:/app/xxx# pip freeze
asn1crypto==0.24.0
bcrypt==3.1.3
blinker==1.4
cachetools==2.0.1
certifi==2018.1.18
cffi==1.11.5
chardet==3.0.4
cryptography==2.1.4
future==0.16.0
google-api-core==0.1.4
google-api-python-client==1.6.4
google-auth==1.4.1
google-auth-httplib2==0.0.2
google-auth-oauthlib==0.1.0
google-cloud-bigquery==0.29.0
google-cloud-core==0.28.1
google-cloud-storage==1.6.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
httplib2==0.10.3
idna==2.6
numpy==1.14.2
oauth2client==4.1.2
oauthlib==2.0.6
pandas==0.22.0
pandas-gbq==0.3.0
paramiko==2.4.1
protobuf==3.5.2
psutil==5.4.3
psycopg2==2.7.4
py4j==0.10.6
pyarrow==0.8.0
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.18
PyHive==0.5.1
PyJWT==1.5.3
pymssql==2.1.4.dev1
PyMySQL==0.8.0
PyNaCl==1.1.2
pyOpenSSL==17.5.0
pysftp==0.2.9
PySocks==1.6.8
pyspark==2.3.0
python-dateutil==2.7.0
pytz==2018.3
PyYAML==3.12
requests==2.18.4
requests-oauthlib==0.8.0
rsa==3.4.2
sasl==0.2.1
scikit-learn==0.19.1
scipy==1.0.0
simplejson==3.13.2
six==1.11.0
SQLAlchemy==1.2.4
thrift==0.11.0
thrift-sasl==0.3.0
uritemplate==3.0.0
urllib3==1.22
Activity
njsmith commentedon Mar 22, 2018
Are you running in some sort of unusual setup, e.g. virtualized? For some reason NumPy has detected that it thinks you have a recent CPU that supports AVX2 instructions, but then when it tries to use those instructions they're causing an "illegal instruction" error.
kparaju commentedon Mar 22, 2018
Yep, it's running inside a docker container inside another VM.
kparaju commentedon Mar 22, 2018
Downgrading to 1.11.3 worked. We will look into if we can not have as many virtualization layers to see if that helps too.
juliantaylor commentedon Mar 25, 2018
That cpu does not support avx2, odd it detects it. Which version of GCC was used to compile numpy?
can you compile and run this file:
I guess we should just parse /proc/cpuinfo to determine avx2 support.
njsmith commentedon Mar 26, 2018
@juliantaylor given all the weird reports we've gotten about different virtualization setups, I almost wonder whether we should test for it by running an instruction and catching the SIGILL (if any).
juliantaylor commentedon Mar 26, 2018
that would be a generic option, though a bit overkill as we currently only support avx with GCC, so linux.
I'll have a look how at how difficult that is to do.
njsmith commentedon Mar 26, 2018
@juliantaylor I guess there's some theoretical problem where if two threads try to use a SIGILL handler like this at once, then they could collide (b/c signal handlers are process-global). This should be extremely rare, of course.
I just reviewed some of the other bug reports we've gotten about this, and you're right, generally
/proc/cpuinfoseems to be correct even withcpuidis broken. So I guess one option would be to figure out how/proc/cpuinfoknows, and do that. And another would be to continue to use__builtin_cpu_supports, but also check/proc/cpuinfoas well (on systems that have it), since this is a weird issue with broken systems, and we're just trying to reduce the frequency with which people hit it.juliantaylor commentedon Mar 27, 2018
the determination is currently during module load which runs under the gil so threads should not be a problem.
My though was to use the builtin and only if it says avx available also check sigill or cpuinfo.
But I am still not really sure why this happens, fftw and openblas also just use cpuid and last I checked I did not see a bug in gccs code.
The only thing I can think if that the OS itself does not support avx, but that means it needs to be something as old as rhel5. Though you can disable it with a boot option, I'll test that.
juliantaylor commentedon Mar 27, 2018
urg, yes disabling avx in the kernel crashes numpy unfortunate gcc does not also verify that in the builtin.
juliantaylor commentedon Mar 27, 2018
fwiw https://gcc.gnu.org/bugzilla/show_bug.cgi?id=85100
BUG: verify the OS supports avx instruction