Skip to content

numpy umath.cpython SIGILL (1.14.2) #10787

Closed
Closed
numpy umath.cpython SIGILL (1.14.2)#10787

Description

@kparaju
kparaju
opened on Mar 22, 2018

Downgrading numpy to 1.11.3 helped

I'm getting a segfault when using numpy 1.14.2. Here's the gdb trace:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `python tests/test_db_bq_loader.py'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x00007fb03004e400 in LONG_add_avx2 (args=<optimized out>,
    dimensions=<optimized out>, steps=<optimized out>,
    __NPY_UNUSED_TAGGEDfunc=<optimized out>)
   from /opt/conda/envs/xx/lib/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so
(gdb)

Here is the code that can replicate this:

import numpy as np
import pandas as pd
from numpy.random import randint
data = [{'ds': '2017-01-01', 'a': 10, 'b': 20, 'c': 30}, {'ds': '2017-01-01', 'a': 11, 'b': 21, 'c': 31},{'ds': '2017-01-01', 'a': 12, 'b': 22, 'c': 32},{'ds': '2017-01-02', 'a': 13, 'b': 23, 'c': 33},{'ds': '2017-01-02', 'a': 14, 'b': 24, 'c': 34},{'ds': '2017-01-03', 'a': 15, 'b': 25, 'c': 35},{'ds': '2017-01-04', 'a': randint(1, 100), 'b': randint(1, 100), 'c': randint(1, 100)}]
df = pd.DataFrame(data)
np.array_equal(df.sort_values(['ds', 'a', 'b', 'c']).values, df.sort_values(['ds', 'a', 'b', 'c']).values)
  • It's running in a docker container. proc/cpuinfo says GenuineIntel Xeon E5-2660

Python version and pip freeze

root@31dedfee1f82:/app/xxx# python --version
Python 3.6.4
root@31dedfee1f82:/app/xxx# pip freeze
asn1crypto==0.24.0
bcrypt==3.1.3
blinker==1.4
cachetools==2.0.1
certifi==2018.1.18
cffi==1.11.5
chardet==3.0.4
cryptography==2.1.4
future==0.16.0
google-api-core==0.1.4
google-api-python-client==1.6.4
google-auth==1.4.1
google-auth-httplib2==0.0.2
google-auth-oauthlib==0.1.0
google-cloud-bigquery==0.29.0
google-cloud-core==0.28.1
google-cloud-storage==1.6.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
httplib2==0.10.3
idna==2.6
numpy==1.14.2
oauth2client==4.1.2
oauthlib==2.0.6
pandas==0.22.0
pandas-gbq==0.3.0
paramiko==2.4.1
protobuf==3.5.2
psutil==5.4.3
psycopg2==2.7.4
py4j==0.10.6
pyarrow==0.8.0
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.18
PyHive==0.5.1
PyJWT==1.5.3
pymssql==2.1.4.dev1
PyMySQL==0.8.0
PyNaCl==1.1.2
pyOpenSSL==17.5.0
pysftp==0.2.9
PySocks==1.6.8
pyspark==2.3.0
python-dateutil==2.7.0
pytz==2018.3
PyYAML==3.12
requests==2.18.4
requests-oauthlib==0.8.0
rsa==3.4.2
sasl==0.2.1
scikit-learn==0.19.1
scipy==1.0.0
simplejson==3.13.2
six==1.11.0
SQLAlchemy==1.2.4
thrift==0.11.0
thrift-sasl==0.3.0
uritemplate==3.0.0
urllib3==1.22

Activity

  1. njsmith commented on Mar 22, 2018

    @njsmith
    njsmith
    on Mar 22, 2018
    Member

    Are you running in some sort of unusual setup, e.g. virtualized? For some reason NumPy has detected that it thinks you have a recent CPU that supports AVX2 instructions, but then when it tries to use those instructions they're causing an "illegal instruction" error.

  2. kparaju commented on Mar 22, 2018

    @kparaju
    kparaju
    on Mar 22, 2018
    Author

    Yep, it's running inside a docker container inside another VM.

  3. kparaju commented on Mar 22, 2018

    @kparaju
    kparaju
    on Mar 22, 2018
    Author

    Downgrading to 1.11.3 worked. We will look into if we can not have as many virtualization layers to see if that helps too.

  4. juliantaylor commented on Mar 25, 2018

    @juliantaylor
    juliantaylor
    on Mar 25, 2018
    Contributor

    That cpu does not support avx2, odd it detects it. Which version of GCC was used to compile numpy?

    can you compile and run this file:

    #include <stdio.h>
int main(void)
{
    printf("avx2 %d\n", __builtin_cpu_supports("avx2"));
}

    gcc file.c
./a.out

    I guess we should just parse /proc/cpuinfo to determine avx2 support.

  5. njsmith commented on Mar 26, 2018

    @njsmith
    njsmith
    on Mar 26, 2018
    Member

    @juliantaylor given all the weird reports we've gotten about different virtualization setups, I almost wonder whether we should test for it by running an instruction and catching the SIGILL (if any).

  6. juliantaylor commented on Mar 26, 2018

    @juliantaylor
    juliantaylor
    on Mar 26, 2018
    Contributor

    that would be a generic option, though a bit overkill as we currently only support avx with GCC, so linux.
    I'll have a look how at how difficult that is to do.

  7. njsmith commented on Mar 26, 2018

    @njsmith
    njsmith
    on Mar 26, 2018
    Member

    @juliantaylor I guess there's some theoretical problem where if two threads try to use a SIGILL handler like this at once, then they could collide (b/c signal handlers are process-global). This should be extremely rare, of course.

    I just reviewed some of the other bug reports we've gotten about this, and you're right, generally /proc/cpuinfo seems to be correct even with cpuid is broken. So I guess one option would be to figure out how /proc/cpuinfo knows, and do that. And another would be to continue to use __builtin_cpu_supports, but also check /proc/cpuinfo as well (on systems that have it), since this is a weird issue with broken systems, and we're just trying to reduce the frequency with which people hit it.

  8. juliantaylor commented on Mar 27, 2018

    @juliantaylor
    juliantaylor
    on Mar 27, 2018
    Contributor

    the determination is currently during module load which runs under the gil so threads should not be a problem.
    My though was to use the builtin and only if it says avx available also check sigill or cpuinfo.

    But I am still not really sure why this happens, fftw and openblas also just use cpuid and last I checked I did not see a bug in gccs code.
    The only thing I can think if that the OS itself does not support avx, but that means it needs to be something as old as rhel5. Though you can disable it with a boot option, I'll test that.

  9. juliantaylor commented on Mar 27, 2018

    @juliantaylor
    juliantaylor
    on Mar 27, 2018
    Contributor

    urg, yes disabling avx in the kernel crashes numpy unfortunate gcc does not also verify that in the builtin.

  10. juliantaylor commented on Mar 27, 2018

    @juliantaylor
    juliantaylor
    on Mar 27, 2018
    Contributor

    fwiw https://gcc.gnu.org/bugzilla/show_bug.cgi?id=85100

  11. charris
    closed this as completedin #10814on Apr 5, 2018
  12. diorcety
    added a commit that references this issue on Jun 9, 2018

    BUG: verify the OS supports avx instruction

    c47819d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions