"Illegal instruction" when running simple array additions #10330

Activity

1. ulido commented on Jan 5, 2018

   

   ulido

   on Jan 5, 2018

   Last edited by ulido

   Author

   More actions

   I just tried that, but it crashed again:

   (master) user@host:~/tmp/numpy$ CFLAGS='-march=x86-64' pip3 install -U .
   Processing /home/ud214/tmp/numpy
   Installing collected packages: numpy
     Found existing installation: numpy 1.15.0.dev0+10ccfe7
       Uninstalling numpy-1.15.0.dev0+10ccfe7:
         Successfully uninstalled numpy-1.15.0.dev0+10ccfe7
     Running setup.py install for numpy ... done
   Successfully installed numpy-1.15.0.dev0+10ccfe7
   You are using pip version 8.1.1, however version 9.0.1 is available.
   You should consider upgrading via the 'pip install --upgrade pip' command.
   (master) user@host:~/tmp$ python3
   Python 3.5.2 (default, Nov 23 2017, 16:37:01)
   [GCC 5.4.0 20160609] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import numpy
   >>> numpy.__version__
   '1.15.0.dev0+10ccfe7'
   >>> a = numpy.arange(15)
   >>> a[1:] + a[:-1]
   Illegal instruction (core dumped)

   I should've mentioned that the problem also arises if I use the regular numpy version from pip:

   (pip) user@host:~/tmp$ pip3 install numpy
   Collecting numpy
     Using cached numpy-1.13.3-cp35-cp35m-manylinux1_x86_64.whl
   Installing collected packages: numpy
   Successfully installed numpy-1.13.3
   You are using pip version 8.1.1, however version 9.0.1 is available.
   You should consider upgrading via the 'pip install --upgrade pip' command.
   (pip) user@host:~/tmp$ python3
   Python 3.5.2 (default, Nov 23 2017, 16:37:01)
   [GCC 5.4.0 20160609] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import numpy
   >>> numpy.__version__
   '1.13.3'
   >>> a = numpy.arange(15)
   >>> a[1:] + a[:-1]
   Illegal instruction (core dumped)

   I guess the pip-supplied manylinux wheel should work regardless of the presence of AVX2, correct?
2. seberg commented on Jan 7, 2018

   

   seberg

   on Jan 7, 2018

   Member

   More actions

   Numpy has runtime detection here. Now that works with gcc __builtin_cpu_supports, either there is something done wrong there in numpy, or the VM is doing something weird? If you have a clue about it, git grep for NPY_CPU_SUPPORTS. GCC docs mention some cpu_init stuff, but dunno, seems weird if that was it?
3. seberg commented on Jan 7, 2018

   

   seberg

   on Jan 7, 2018

   Member

   More actions

   I am sure you can remove the avx/avx2 target from the setup code (or pass through flags to disable it maybe). (e.g. check/modify numpy/core/setup_common.py)
4. ulido commented on Jan 7, 2018

   

   ulido

   on Jan 7, 2018

   Author

   More actions

   Thanks @seberg ! I'm pretty sure that the problem is that the Xen hypervisor my VM is running on is configure to mask the AVX capabilities of the host cpu (I don't know why, I don't control the hypervisor.). My guess is that __builtin_cpu_supports uses heuristics based on the cpu name or so to find which features it supports. However, that leads to trouble in this case when capabilities are masked/forbidden by the host. Here is some code to prove that (AVX is detected but not working):

   user@host:~/tmp$ cat avx_test.c
   #include <stdio.h>
   
   int main()
   {
     if (__builtin_cpu_supports("avx")) {
       printf("AVX supported!\n");
   
       printf("Testing if it's really working (should crash with SIGILL if not)\n");
       // Simplest AVX instruction I could find:
       __builtin_ia32_vzeroall();
       printf("If you see this, it worked!\n");
     } else {
       printf("AVX not supported!\n");
     }
   }
   user@host:~/tmp$ gcc -mavx avx_test.c
   user@host:~/tmp$ ./a.out
   AVX supported!
   Testing if it's really working (should crash with SIGILL if not)
   
   Illegal instruction (core dumped)
   

   Do you think it would be possible to use a custom function to set NPY_CPU_SUPPORTS_AVX / NPY_CPU_SUPPORTS_AVX2 that tries to run a corresponding instruction and catches SIGILL instead of __builtin_cpu_supports?
5. seberg commented on Jan 7, 2018

   

   seberg

   on Jan 7, 2018

   Member

   More actions

   As far as I understand AVX, etc. have to be enabled explicitly when compiling? So I think you can simply remove those lines that have something to do with AVX in that setup file I mentioned and recompile? That should just lead to disabling the whole stuff.

   For a more general solution, I do not know.... I am sure we could have a flag or so to disable it explicitly at least during compile time, maybe even more fancy stuff. I do not really know build stuff, etc. Doing more fancy stuff during run-time seems odd to me, but thats just gut feeling.
6. ulido commented on Jan 7, 2018

   

   ulido

   on Jan 7, 2018 via email

   Author

   More actions

   Yes, I can disable AVX at compile time, that wouldn't be a problem. However that doesn't fix it for an unsuspecting user who tries to `pip install numpy` in such a VM and ends up with a cryptic "Illegal Instruction" error. I'll investigate a bit further what could be done.

   …

   On Sun, 7 Jan 2018, 17:30 seberg, ***@***.***> wrote: As far as I understand AVX, etc. have to be enabled explicitly when compiling? So I think you can simply remove those lines that have something to do with AVX in that setup file I mentioned and recompile? That should just lead to disabling the whole stuff. For a more general solution, I do not know.... I am sure we could have a flag or so to disable it explicitly at least during compile time, maybe even more fancy stuff. I do not really know build stuff, etc. Doing more fancy stuff during run-time seems odd to me, but thats just gut feeling. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#10330 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB40auCRLUYdCKy3RJOdtMpY7ckQ7Em_ks5tIP8ogaJpZM4RUpgn> .
8. seberg commented on Jan 7, 2018

   

   seberg

   on Jan 7, 2018

   Member

   More actions

   Tend to agree, it seems more like a bug in either gcc or probably the VM to me as well. I don't mind making working around it easier, but doing weird things when the gcc function seems like smarter people who know this stuff wrote it….
   We can make working around it easier maybe. I also wonder if (at least for manylinux numpy) the openblas won't blow up as well as soon as you use it? And I am sure adding that __builtin_cpu_init() call does not matter?
9. ulido commented on Jan 8, 2018

   

   ulido

   on Jan 8, 2018

   Author

   More actions

   Ok, @seberg, I followed your suggestion and removed the entries for AVX in setup_common.py, which fixed the problem for now, but I guess it would be good to at least have a compile time option to disable AVX/AVX2.

   I did some testing and found that it is not a gcc bug. The cpu itself reports AVX/AVX2 capabilities via the cpuid instruction (which is used by __builtin_cpu_supports). Rather, Xen fails to mask the features in the response to calling cpuid (but I don't know anything about Xen, and have no control over the host system I'm running my VM on). /proc/cpuinfo correctly reports them as absent, so it should be possible to query that as a last check after __builtin_cpu_supports reports true.

   Closing the issue for now.

   👍React with 👍1Reacted by Seokjoon Ahn
10. ulido

    closed this as completedon Jan 8, 2018