Skip to content

numpy 1.25.0 fails with illegal instruction in aws batch #24028

Closed
#24311
Closed
numpy 1.25.0 fails with illegal instruction in aws batch#24028
#24311
Labels
32 - InstallationProblems installing or compiling NumPy

Description

@emilyselwood
emilyselwood
opened on Jun 23, 2023

Steps to reproduce:

import numpy 1.25.0 in a task running in aws-batch fails with illegal instruction but runs locally.

python version: 3.10.11
working numpy version: 1.24.3
not working numpy version: 1.25.0

AWS batch is configured to use the following instance types:

  • m6g
  • r6g
  • c7g

Error message:

bash: line 1: 47 Illegal instruction (core dumped) ( python -u src-metrics-flows-test.py --quiet --metadata se

Unfortunately due to running in AWS batch we are not able to get the core dump currently.

Additional information:

Roll back to 1.24.3 and it works again without us changing any code. I would hazard an un-scientific guess that this could be related to the build work thats been going on.

Activity

  1. emilyselwood
    added
    32 - InstallationProblems installing or compiling NumPy
    on Jun 23, 2023

  2. seberg commented on Jun 23, 2023

    @seberg
    seberg
    on Jun 23, 2023
    Member

    Can you help us by saying what type of machines these are? I guess graviton? Also, getting the output of np.show_runtime() is likely necessary to narrow this down.

    How exactly did you install/build numpy?

    this could be related to the build work thats been going on.

    The build work shouldn't affect the released versions of NumPy, although it could be a build issue. Another possibility is incorrect runtime CPU detection, that sometimes happens in virtual machines for example.

  3. emilyselwood commented on Jun 27, 2023

    @emilyselwood
    emilyselwood
    on Jun 27, 2023
    Author

    A quick update, it seems to be the 7g series that cause the problem, but due to having both 6g and 7g nodes available it has become intermittent triggering this one.

    I'll update with the results from a c7g node when I can get them

  4. seberg commented on Jun 27, 2023

    @seberg
    seberg
    on Jun 27, 2023
    Member

    Thanks, in that case, it may be good to see the difference between the two also (if there is any).

  5. james-takeone commented on Jul 13, 2023

    @james-takeone
    james-takeone
    on Jul 13, 2023

    I am seeing what I believe to be the same issue. I'm also using c7g instances on AWS Batch (EDIT: uses Graviton3). I upgraded from 1.24.3 to 1.25.1 today and all my AWS Batch jobs are failing with exit code 132, which based on some searching seems like is related to invalid instructions. Reverting to 1.24.3 resolved the issue. I instrumented my application to add the following to the entrypoint:

    import numpy as np
np.show_runtime()

    For context, I'm installing numpy via pip as part of the docker image build process (i.e., I don't build numpy from source). If it's helpful, the docker image is built on an M1 Macbook Pro running Docker Engine v24.0.2.

    For 1.24.3 running on c7g instances on AWS Batch, I get the following output:

    [{'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
'found': ['ASIMDHP', 'ASIMDDP', 'ASIMDFHM'],
'not_found': []}},
{'architecture': 'neoversev1',
'filepath': '/var/task/numpy.libs/libopenblas64_p-r0-cecebdce.3.21.so',
'internal_api': 'openblas',
'num_threads': 8,
'prefix': 'libopenblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.21'}]

    For 1.25.1 c7g instances on AWS Batch, the container exits with code 132 prior to printing anything.

    For 1.24.3 running on my M1 Macbook Pro, I get the following output:

    [{'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP', 'ASIMDDP', 'ASIMDFHM'],
                      'not_found': []}},
 {'architecture': 'armv8',
  'filepath': '/var/task/numpy.libs/libopenblas64_p-r0-cecebdce.3.21.so',
  'internal_api': 'openblas',
  'num_threads': 8,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.21'}]

    For 1.25.1 running on my M1 Macbook Pro, I get the following output:

    [{'numpy_version': '1.25.1',
  'python': '3.10.12 (main, Jul  4 2023, 22:17:40) [GCC 10.2.1 20210110]',
  'uname': uname_result(system='Linux', node='4c76e3da03ce', release='5.15.49-linuxkit-pr', version='#1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023', machine='aarch64')},
 {'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP', 'ASIMDDP', 'ASIMDFHM'],
                      'not_found': []}},
 {'architecture': 'armv8',
  'filepath': '/var/task/numpy.libs/libopenblas64_p-r0-9fed50d4.3.23.so',
  'internal_api': 'openblas',
  'num_threads': 8,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.23'}]

  6. mattip commented on Jul 13, 2023

    @mattip
    mattip
    on Jul 13, 2023
    Member

    There are two unrelated systems to use advanced SIMD instructions in NumPy. One is builtin, and is reported in the simd_extensions section. The other is via OpenBLAS, which has its own runtime dispatch and is reflected in the architecture section.

    Could you try limiting the simd_extension runtime dispatch by adding export NPY_DISABLE_CPU_FEATURES="ASIMDHP, ASIMDDP,ASIMDFHM" to the image? If you do this with the successful run of 1.24.3, you should see that change reflected in the simd_extensions entry of the np.show_runtime() output.

    If that does not work, the problem might be with OpenBLAS (although I think this is less likely). Its dispatch can be controlled by adding export OPENBLAS_CORETYPE=ARMV8. If you do this with the successful run of 1.24.3, you should see that change reflected in the architecture entry of the np.show_runtime() output.

  7. james-takeone commented on Jul 13, 2023

    @james-takeone
    james-takeone
    on Jul 13, 2023

    Quickly spun up a c7g ec2 instance to test outside of a docker container. Test consisted of creating a virtual env, and installing numpy, and running the python repl.

    Here's the output for 1.24.3:

    >>> import sys, numpy; print(numpy.__version__); print(sys.version)
1.24.3
3.9.16 (main, Feb 23 2023, 00:00:00) 
[GCC 11.3.1 20221121 (Red Hat 11.3.1-4)]
>>> numpy.show_runtime()
[{'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP', 'ASIMDDP', 'ASIMDFHM'],
                      'not_found': []}},
 {'architecture': 'neoversev1',
  'filepath': '/home/ec2-user/numpy-test/myenv/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-cecebdce.3.21.so',
  'internal_api': 'openblas',
  'num_threads': 2,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.21'}]

    Here's the output for 1.25.1:

    >>> import sys, numpy; print(numpy.__version__); print(sys.version)
1.25.1
3.9.16 (main, Feb 23 2023, 00:00:00) 
[GCC 11.3.1 20221121 (Red Hat 11.3.1-4)]
>>> numpy.show_runtime()
[{'numpy_version': '1.25.1',
  'python': '3.9.16 (main, Feb 23 2023, 00:00:00) \n'
            '[GCC 11.3.1 20221121 (Red Hat 11.3.1-4)]',
  'uname': uname_result(system='Linux', node='ip-10-0-0-191.ec2.internal', release='6.1.34-59.116.amzn2023.aarch64', version='#1 SMP Thu Jun 29 18:11:49 UTC 2023', machine='aarch64')},
 {'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP', 'ASIMDDP', 'ASIMDFHM'],
                      'not_found': []}},
 {'architecture': 'neoversev1',
  'filepath': '/home/ec2-user/numpy-test/myenv/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-9fed50d4.3.23.so',
  'internal_api': 'openblas',
  'num_threads': 2,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.23'}]

  8. james-takeone commented on Jul 13, 2023

    @james-takeone
    james-takeone
    on Jul 13, 2023

    Thanks @mattip, I'll give that a shot tomorrow and report back.

  9. mattip commented on Jul 13, 2023

    @mattip
    mattip
    on Jul 13, 2023
    Member

    test outside of a docker container

    Hmm. Outside the container, you can use NumPy 1.25.1. Inside the container, you get an illegal instruction. Perhaps you are pulling in the macos wheel for numpy when building the container, instead of the linux aarch64 one?

  10. james-takeone commented on Jul 13, 2023

    @james-takeone
    james-takeone
    on Jul 13, 2023

    Hmm. Outside the container, you can use NumPy 1.25.1. Inside the container, you get an illegal instruction. Perhaps you are pulling in the macos wheel for numpy when building the container, instead of the linux aarch64 one?

    I'm not sure how to check that. However, I was able to run the same image directly on the c7g EC2 instance that fails on AWS Batch. Here is the output:

    >>> import sys, numpy; print(numpy.__version__); print(sys.version)
1.25.1
3.10.12 (main, Jul  4 2023, 22:17:40) [GCC 10.2.1 20210110]
>>> numpy.show_runtime()
[{'numpy_version': '1.25.1',
  'python': '3.10.12 (main, Jul  4 2023, 22:17:40) [GCC 10.2.1 20210110]',
  'uname': uname_result(system='Linux', node='b4192cb08e0b', release='6.1.34-59.116.amzn2023.aarch64', version='#1 SMP Thu Jun 29 18:11:49 UTC 2023', machine='aarch64')},
 {'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP', 'ASIMDDP', 'ASIMDFHM'],
                      'not_found': []}},
 {'architecture': 'neoversev1',
  'filepath': '/var/task/numpy.libs/libopenblas64_p-r0-9fed50d4.3.23.so',
  'internal_api': 'openblas',
  'num_threads': 2,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.23'}]

    I'm a bit out of my depth now, so I'm going to reach out to AWS business support for some assistance. Thanks for your help @mattip!

  11. martin-frbg commented on Jul 18, 2023

    @martin-frbg
    martin-frbg
    on Jul 18, 2023

    It occurs to me that we (OpenBLAS) are naively expecting SVE to be available if the detected cpu is known to support it (there's currently no fine-grained checking of hwcap, unlike other platforms) , maybe that is not always a given with containers ?

  12. james-takeone commented on Jul 18, 2023

    @james-takeone
    james-takeone
    on Jul 18, 2023

    @martin-frbg Is there anything that I can do / add to our container to help debug this?

  13. martin-frbg commented on Jul 18, 2023

    @martin-frbg
    martin-frbg
    on Jul 18, 2023

    If you could add a call to lscpu or something to print the contents of /proc/cpuinfo (if available) to see if "sve" shows up among the feature flags that would be great

  14. james-takeone commented on Jul 18, 2023

    @james-takeone
    james-takeone
    on Jul 18, 2023

    @martin-frbg I was able to run the following directly on the host as well as instrument the container to run the following on entry.

    TL;DR
    It looks like your hunch was correct, SVE does **not ** appear in the AWS Batch processor feature list, although it does appear when running the commands directly on the host as well as in the image running on Docker.

    Note that the same image is used when testing Docker via running on the host C7G instance and AWS Batch.

    echo "****** Printing Diagnostic Info *****"
echo "****** start lscpu *****"
lscpu
echo "****** end lscpu *****"
echo "****** start cat /proc/cpuinfo *****"
cat /proc/cpuinfo
echo "****** end cat /proc/cpuinfo *****"

    Here are the results:

    Directly on the host C7G EC2 instance:

    ****** Printing Diagnostic Info *****
****** start lscpu *****
Architecture:          aarch64
  CPU op-mode(s):      32-bit, 64-bit
  Byte Order:          Little Endian
CPU(s):                2
  On-line CPU(s) list: 0,1
Vendor ID:             ARM
  Model:               1
  Thread(s) per core:  1
  Core(s) per socket:  2
  Socket(s):           1
  Stepping:            r1p1
  BogoMIPS:            2100.00
  Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc 
                       dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei
                       8mm svebf16 i8mm bf16 dgh rng
Caches (sum of all):   
  L1d:                 128 KiB (2 instances)
  L1i:                 128 KiB (2 instances)
  L2:                  2 MiB (2 instances)
  L3:                  32 MiB (1 instance)
NUMA:                  
  NUMA node(s):        1
  NUMA node0 CPU(s):   0,1
Vulnerabilities:       
  Itlb multihit:       Not affected
  L1tf:                Not affected
  Mds:                 Not affected
  Meltdown:            Not affected
  Mmio stale data:     Not affected
  Retbleed:            Not affected
  Spec store bypass:   Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:          Mitigation; __user pointer sanitization
  Spectre v2:          Mitigation; CSV2, BHB
  Srbds:               Not affected
  Tsx async abort:     Not affected
****** end lscpu *****
****** start cat /proc/cpuinfo *****
processor       : 0
BogoMIPS        : 2100.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0xd40
CPU revision    : 1

processor       : 1
BogoMIPS        : 2100.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0xd40
CPU revision    : 1

****** end cat /proc/cpuinfo *****

    Docker running on the C7G EC2 Instance:

    ****** Printing Diagnostic Info *****
****** start lscpu *****
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       ARM
Model:                           1
Stepping:                        r1p1
BogoMIPS:                        2100.00
L1d cache:                       128 KiB
L1i cache:                       128 KiB
L2 cache:                        2 MiB
L3 cache:                        32 MiB
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; CSV2, BHB
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
****** end lscpu *****
****** start cat /proc/cpuinfo *****
processor       : 0
BogoMIPS        : 2100.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0xd40
CPU revision    : 1

processor       : 1
BogoMIPS        : 2100.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0xd40
CPU revision    : 1

****** end cat /proc/cpuinfo *****

    AWS Batch running on a C7G instance:

    ****** Printing Diagnostic Info *****
****** start lscpu *****
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       ARM
Model:                           1
Stepping:                        r1p1
BogoMIPS:                        2100.00
L1d cache:                       128 KiB
L1i cache:                       128 KiB
L2 cache:                        2 MiB
L3 cache:                        32 MiB
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; CSV2, BHB
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs rng
****** end lscpu *****
****** start cat /proc/cpuinfo *****
processor	: 0
BogoMIPS	: 2100.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs rng
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x1
CPU part	: 0xd40
CPU revision	: 1
processor	: 1
BogoMIPS	: 2100.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs rng
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x1
CPU part	: 0xd40
CPU revision	: 1
****** end cat /proc/cpuinfo *****

  15. martin-frbg commented on Jul 19, 2023

    @martin-frbg
    martin-frbg
    on Jul 19, 2023

    That would mean you could only run with OPENBLAS_CORETYPE=NEOVERSEN1 on AWS Batch, unless AWS support can
    tell if/how to enable SVE with AWS Batch. (Without SVE enabled, you might as well use one of the Graviton2 instances which are possibly cheaper)

  16. 29 remaining items

  17. roypat
    added 15 commits that reference this issue on Sep 14, 2023

    chore: Rebuild devctr

    7bafccd

    chore: Rebuild devctr

    7ece53e

    chore: Rebuild devctr

    87ed346

    chore: Rebuild devctr

    fbde946

    chore: Rebuild devctr

    44cdfa5

    chore: Rebuild devctr

    655dff3

    chore: Rebuild devctr

    88ea0aa

    chore: Rebuild devctr

    9aea098

    chore: Rebuild devctr

    6363764

    chore: Rebuild devctr

    d03413d

    chore: Rebuild devctr

    98daa3b

    chore: Rebuild devctr

    2835dc1

    chore: Rebuild devctr

    767a9ca

    chore: Rebuild devctr

    f5ff601

    chore: Rebuild devctr

    8d6334d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    32 - InstallationProblems installing or compiling NumPy

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions