Repository navigation
Numpy crashes during tests with a "Illegal instruction (core dumped)". #9532
Description
Hello,
I've run into a really cryptic problem with numpy's functionality on a virtual machine running Ubuntu 16.04 with ~2GB of RAM (just a tad less) and 100MB of video memory on Python2.7. The virtual machine was done on my laptop and then exported. It was tested on another laptop where it worked. When we tried running it on third laptop it kept crashing. The laptop configuration is:
OS Name Microsoft Windows 10 Pro
Version 10.0.14393 Build 14393
Other OS Description Not Available
OS Manufacturer Microsoft Corporation
System Name DESKTOP-N693E6G
System Manufacturer LENOVO
System Model 80EW
System Type x64-based PC
System SKU LENOVO_MT_80EW_BU_idea_FM_Lenovo B50-80
Processor Intel(R) Core(TM) i3-5005U CPU @ 2.00GHz, 2000 Mhz, 2 Core(s), 4 Logical Processor(s)
BIOS Version/Date LENOVO A8CN51WW(V3.04), 26.2.2016.
SMBIOS Version 2.7
Embedded Controller Version 3.51
BIOS Mode Legacy
BaseBoard Manufacturer LENOVO
BaseBoard Model Not Available
BaseBoard Name Base Board
Platform Role Mobile
Secure Boot State Unsupported
PCR7 Configuration Binding Not Possible
Windows Directory C:\WINDOWS
System Directory C:\WINDOWS\system32
Boot Device \Device\HarddiskVolume1
Locale Hrvatska
Hardware Abstraction Layer Version = "10.0.14393.1378"
User Name DESKTOP-N693E6G\Korisnik
Time Zone Central European Daylight Time
Installed Physical Memory (RAM) 4,00 GB
Total Physical Memory 3,92 GB
Available Physical Memory 1,30 GB
Total Virtual Memory 5,79 GB
Available Virtual Memory 1,95 GB
Page File Space 1,88 GB
Page File C:\pagefile.sys
Hyper-V - VM Monitor Mode Extensions Yes
Hyper-V - Second Level Address Translation Extensions Yes
Hyper-V - Virtualization Enabled in Firmware No
Hyper-V - Data Execution Protection Yes
The error manifests itself as an Illegal instruction (core dumped) and the shell restarts, or if you ran the program from the terminal you get back to a clean terminal line. Eventually the error was tracked to a np.where command which seems to reproduce the behavior consistently. I've used the following code to test it:
student@vsa:~/Desktop/PracticeExamples/6_contours_fitObj$ cat test.py
import numpy as np
a = np.arange(0, 100, 1)
np.where(a>90)
When I run it with gdb I get:
student@vsa:~/Desktop/PracticeExamples/6_contours_fitObj$ gdb -ex r --args python test.py
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...(no debugging symbols found)...done.
Starting program: /usr/bin/python test.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Program received signal SIGILL, Illegal instruction.
0xb61338a8 in LONG_greater_avx2 (args=args@entry=0xbfffe52c,
dimensions=dimensions@entry=0xbfffe538, steps=steps@entry=0xbfffe544,
__NPY_UNUSED_TAGGEDfunc=__NPY_UNUSED_TAGGEDfunc@entry=0x0)
at numpy/core/src/umath/loops.c.src:936
936 numpy/core/src/umath/loops.c.src: No such file or directory.
(gdb) bt
#0 0xb61338a8 in LONG_greater_avx2 (args=args@entry=0xbfffe52c,
dimensions=dimensions@entry=0xbfffe538, steps=steps@entry=0xbfffe544,
__NPY_UNUSED_TAGGEDfunc=__NPY_UNUSED_TAGGEDfunc@entry=0x0)
at numpy/core/src/umath/loops.c.src:936
#1 0xb617d5a6 in trivial_three_operand_loop (op=op@entry=0xbfffe8f0,
innerloop=0xb61337ac <LONG_greater_avx2>, innerloopdata=0x0)
at numpy/core/src/umath/ufunc_object.c:1339
#2 0xb61856d7 in execute_legacy_ufunc_loop (arr_prep_args=0x0, arr_prep=0xbfffe680,
buffersize=8192, order=NPY_KEEPORDER, dtypes=0xbfffe600, op=0xbfffe8f0,
trivial_loop_ok=<optimized out>, ufunc=0x84eadb8) at numpy/core/src/umath/ufunc_object.c:1700
#3 PyUFunc_GenericFunction (ufunc=ufunc@entry=0x84eadb8, args=args@entry=0xb7cda78c,
kwds=kwds@entry=0x0, op=op@entry=0xbfffe8f0) at numpy/core/src/umath/ufunc_object.c:2749
#4 0xb61869c0 in ufunc_generic_call (ufunc=0x84eadb8, args=0xb7cda78c, kwds=0x0)
at numpy/core/src/umath/ufunc_object.c:4411
#5 0x080f90d9 in PyObject_Call ()
#6 0x081017e9 in PyObject_CallFunctionObjArgs ()
#7 0xb79a2dd7 in PyArray_GenericBinaryFunction (m1=0xb7a73a98, m2=0x8412c3c, op=0x84eadb8)
at numpy/core/src/multiarray/number.c:260
#8 0x080f4f1c in PyObject_RichCompare ()
#9 0x08111a00 in PyEval_EvalFrameEx ()
#10 0x0810a47b in PyEval_EvalCodeEx ()
#11 0x0810a26e in PyEval_EvalCode ()
#12 0x08139418 in ?? ()
#13 0x08133ea5 in PyRun_FileExFlags ()
#14 0x08132838 in PyRun_SimpleFileExFlags ()
#15 0x08132684 in PyRun_AnyFileExFlags ()
#16 0x080e65bc in Py_Main ()
#17 0x080e5ec6 in main ()
(gdb) p a
No symbol "a" in current context.
(gdb) u
Program terminated with signal SIGILL, Illegal instruction.
The program no longer exists.
(gdb)
not sure if relevant but numpy also consistently fails to survive the tests on the buss dates:
student@vsa:~/Desktop/PracticeExamples/6_contours_fitObj$ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.13.1'
>>> numpy.__file__
'/usr/local/lib/python2.7/dist-packages/numpy/__init__.pyc'
>>> numpy.test(verbose=10)
Running unit tests for numpy
NumPy version 1.13.1
NumPy relaxed strides checking option: True
NumPy is installed in /usr/local/lib/python2.7/dist-packages/numpy
Python version 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
nose version 1.3.7
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
test_complex (test_abc.ABC) ... ok
test_floats (test_abc.ABC) ... ok
test_int (test_abc.ABC) ... ok
test_uint (test_abc.ABC) ... ok
test_api.test_array_array ... ok
test_api.test_fastCopyAndTranspose ... ok
test_api.test_array_astype ... ok
test_api.test_copyto_fromscalar ... ok
test_api.test_copyto ... ok
test_api.test_copyto_permut ... ok
test_api.test_copy_order ... ok
test_api.test_contiguous_flags ... ok
test_api.test_broadcast_arrays ... ok
Basic test of array2string. ... ok
Test custom format function for each element in array. ... ok
test_structure_format (test_arrayprint.TestArray2String) ... ok
This should only apply to 0-D arrays. See #1218. ... ok
test_arrayprint.TestArrayRepr.test_containing_list ... ok
test_arrayprint.TestArrayRepr.test_nan_inf ... ok
test_arrayprint.TestArrayRepr.test_self_containing ... ok
test_arrayprint.TestArrayRepr.test_subclass ... ok
test_str (test_arrayprint.TestComplexArray) ... ok
test_arrayprint.TestPrintOptions.test_basic ... ok
test_arrayprint.TestPrintOptions.test_formatter ... ok
test_arrayprint.TestPrintOptions.test_formatter_reset ... ok
test_arrayprint.TestPrintOptions.test_precision_zero ... ok
test_arrayprint.test_unicode_object_array ... ok
test_cast_overflow (test_datetime.TestDateTime) ... ok
test_compare_generic_nat (test_datetime.TestDateTime) ... ok
test_creation_overflow (test_datetime.TestDateTime) ... ok
test_datetime_add (test_datetime.TestDateTime) ... ok
test_datetime_arange (test_datetime.TestDateTime) ... ok
test_datetime_arange_no_dtype (test_datetime.TestDateTime) ... ok
test_datetime_array_find_type (test_datetime.TestDateTime) ... ok
test_datetime_array_str (test_datetime.TestDateTime) ... ok
test_datetime_as_string (test_datetime.TestDateTime) ... ok
test_datetime_as_string_timezone (test_datetime.TestDateTime) ... ok
test_datetime_busday_holidays_count (test_datetime.TestDateTime) ... Illegal instruction (core dumped)
student@vsa:~/Desktop/PracticeExamples/6_contours_fitObj$
The plan is to use the virtual machines on a summer school to avoid having to setup the whole machine for everyone individually which would include setting up the virtual machine, installing Ubuntu on it, installing numpy, scipy, fitsio and then compiling OpenCV 3.0. Some help would be greatly appreciated.
Thanks!
Activity
juliantaylor commentedon Aug 8, 2017
the virtual machine probably exports cpu capabilities your real cpu does not have.
what is /proc/cpuinfo on the host and the guest?
juliantaylor commentedon Aug 8, 2017
hm the host is windows? that cpu should have avx2 so maybe the vm is configured to not support it but in a way numpy does not detect (numpy essentially uses cpuid)
juliantaylor commentedon Aug 8, 2017
In the guest avx2 is not listed in the flags, that is a misconfiguration when cpuid still returns it.
Might make sense to put an extra check into numpy for wrongly configured vms
juliantaylor commentedon Aug 8, 2017
depends on the vm software used, there should be a way to change the cpu features the guest has somewhere in it.
njsmith commentedon Aug 8, 2017
I wonder how Linux is detecting avx2 if not via cpuid...
juliantaylor commentedon Aug 8, 2017
Reinstalling numpy won't change anything, numpy tries to detect avx2 at runtime via the cpuid instruction.
It is strange that the VM sigills the instruction but still returns essentially wrong results with cpuid. Maybe the used VM software just doesn't support avx2 at all.
njsmith commentedon Aug 8, 2017
@ljetibo: @juliantaylor's theory is that numpy is like "hey, virtual CPU, do you support AVX2?" and it's like "yep, I sure do!" and then numpy's like "okay, here's some AVX2 instructions to execute!" and the virtual CPU is like "omgwtf are these I never heard of this AVX2 thing". This can happen if the VM hypervisor is buggy. (What hypervisor are you using btw?)
@juliantaylor: I guess one way to eliminate this class of bugs entirely would be to make our AVX2 detection logic be: (1) try to execute some AVX2, (2) if we get SIGILL, then the CPU doesn't support AVX2. Making this actually work would require some gross stuff with signal handlers and longjmp though, plus I don't know, SEH or something on Windows? And we'd get some very confused bug reports by people trying to debug something else using gdb and seeing a SIGILL and freaking out. OTOH, "actually works" is a nice feature.
bashtage commentedon Aug 9, 2017
One solution would be to create your VM on a old CPU that doesn't have AVX2 (or possibly AVX, if you want maximum portability).
bashtage commentedon Aug 9, 2017
Also, that the
Intel(R) Core(TM) i3-5005U CPU @ 2.00GHzlaptop doesn't report AVX support, even though it should (http://ark.intel.com/products/84695/Intel-Core-i3-5005U-Processor-3M-Cache-2_00-GHz), suggests that laptop's OS is mis-configured, since OS support is required to use AVX.bashtage commentedon Aug 9, 2017
I don't think setting up student PCs using Anaconda or Miniconda is particularly hard. You could easily provide a script that would do all of the work, even pinning packages if you need to.
juliantaylor commentedon Aug 9, 2017
to remove avx use you currently have to patch
numpy/core/include/numpy/npy_common.h, the lines with AVX in them, e.g. just set all cases to 0.An environment variable to switch cpu instruction sets without recompile might be a good idea for the future.
snig-b commentedon Aug 30, 2017
I ran into the same issue. I'm using a virtual environment but not a virtualbox, only virtualenv. The OS config remains the same. I have the following OS config for Ubuntu 16.04 (it has the avx flag):
And the following on a separate laptop - mac:
On which it works
Incidently an np.array comparison
np.array(range(0,31)) < 0works for upto this length of the list, but all other lengths greater than this fail with Illegal Instruction.
Any help regarding this is appreciated.
juliantaylor commentedon Sep 1, 2017
that is very odd, so you are using real hardware no virtual machine and it still crashes? virtualenv has nothing to do with virtual machines.
That should not happen was your machine does support avx2 according to the cpuinfo.
Can you run this script on your machine?
Put into a .c file and compile with
gcc -mavx2 file.cand run the resulting./a.outfile and just see if it crashes or not.snig-b commentedon Sep 3, 2017
I ran the test, and yeah it is crashing.
What is missing in the system?
juliantaylor commentedon Sep 3, 2017
Odd that cpu is supposed to have avx2 according to intel and the OS is definitely new enough to support it. I am not aware of any method to disable the instructions and keep the kernel reporting it available.
So I do not know what is wrong.
mknorps commentedon Jan 29, 2018
I have the same problem on my VM. As @juliantaylor suggested, I changed lines with AVX in them to 0 in numpy/core/include/numpy/npy_common.h, but I have problem with building numpy with those changes:
Can you help me?
The setup of my virtual environment:
EDIT:
I needed to download module 'numpy._build_utils' from numpy/numpy github code, because it is not included in pip release. Now
works,
but even after changing lines with AVX in to 0 the program test.py is crashing.
njsmith commentedon Jan 29, 2018
krassowski commentedon Jan 29, 2018
@mknorps, if I remember correctly, this worked for me:
This assumes that you have standard build tools and python3-dev installed.
For more detailed instructions refer to https://docs.scipy.org/doc/numpy/user/building.html
Just to prevent any doubts, here is my git diff:
mknorps commentedon Jan 30, 2018
Thank you for your help, this solved this issue indeed.
Unfortunately after restart my virtual machine does not boot, but this is whole another issue with Virtual Box, specificaly with running 32 bit guest system on 64 bit host.
pcordes commentedon Dec 8, 2022
BTW, disabling AVX support in NumPY is a backwards approach if your CPU does support AVX. AVX (and FMA3) are quite helpful for throughput in some types of problems, so you want to use it. Fix your VM, and/or your ancient guest OS, so AVX works properly.
No, it doesn't. That
/proc/cpuinfoincludesavx2but notavx. So perhaps that's why the guest kernel wouldn't set the control-register bit to allow AVX1/2 instruction to run without faulting, if it was just looking for the base AVX feature bit. (That's the one that matters for context switches saving full YMM registers).A CPUID that reports AVX2 but not AVX1 is a broken config, only possible in a VM (since the
cpuidinstruction is always a vm-exit, so the VM can tweak what feature flags and other info the guest sees).Related:
__builtin_cpu_supports("avx")and"avx2"checks for OS support as well.NumPY uses
cpuidin the guest (via GCC's builtin) to check for AVX2 and AVX. (And FMA3).IDK why people would be getting crashes with a non-ancient OS like Ubuntu 16.04, unless a VM is screwing up XCR0 settings, if the VM exposes AVX feature bits to the guest.