Skip to content

Failed to build v1.17.0 on CentOS7 with default gcc (4.8) #14147

Closed

Description

@xkszltl

Cannot build numpy v1.17.0 on CentOS 7.
Previous version works fine.
I'm seeing a lot of C99/C11 features in error message, and avx512f.
Not sure if the required --std option is set but I don't think gcc 4.8.5 (the system default compiler on CentOS 7) can handle all of that.

Reproducing code example:

Simply build the wheel on CentOS 7 with python3.

Error message:

Some error log attached.
The entire story is too long to be pasted here.

#6 356.1   compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -c'
#6 356.1   gcc: _configtest.c
#6 356.1   _configtest.c: In function 'main':
#6 356.1   _configtest.c:5:3: warning: implicit declaration of function '__builtin_mul_overflow' [-Wimplicit-function-declaration]
#6 356.1      __builtin_mul_overflow(5, 5, (int*)5);
#6 356.1      ^
#6 356.1   gcc -pthread _configtest.o -o _configtest
#6 356.1   _configtest.o: In function `main':
#6 356.1   /tmp/pip-req-build-poiltggx/_configtest.c:5: undefined reference to `__builtin_mul_overflow'
#6 356.1   collect2: error: ld returned 1 exit status
#6 356.1   failure.
#6 356.1   removing: _configtest.c _configtest.o _configtest.o.d
#6 356.1   C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC
#6 356.1   
#6 356.1   compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -c'
#6 356.1   gcc: _configtest.c
#6 356.1   _configtest.c: In function 'main':
#6 356.1   _configtest.c:7:16: warning: unused variable 'r' [-Wunused-variable]
#6 356.1      volatile int r = __builtin_cpu_supports("sse");
#6 356.1                   ^
#6 356.1   gcc -pthread _configtest.o -o _configtest
#6 356.1   success!
#6 356.1   removing: _configtest.c _configtest.o _configtest.o.d _configtest
#6 356.1   C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC
#6 356.1   
#6 356.1   compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -c'
#6 356.1   gcc: _configtest.c
#6 356.1   _configtest.c: In function 'main':
#6 356.1   _configtest.c:7:3: error: Parameter to builtin not valid: avx512f
#6 356.1      volatile int r = __builtin_cpu_supports("avx512f");
#6 356.1      ^
#6 356.1   _configtest.c:7:16: warning: unused variable 'r' [-Wunused-variable]
#6 356.1      volatile int r = __builtin_cpu_supports("avx512f");
#6 356.1                   ^
#6 356.1   failure.
#6 356.1   removing: _configtest.c _configtest.o
#6 356.1   C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC
#6 356.1   
#6 356.1   compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -c'
#6 356.1   gcc: _configtest.c
#6 356.1   gcc -pthread _configtest.o -o _configtest
#6 356.1   success!
#6 356.1   removing: _configtest.c _configtest.o _configtest.o.d _configtest
#6 356.1   C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC
#6 356.1   
#6 356.1   compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -c'
#6 356.1   gcc: _configtest.c
#6 356.1   gcc -pthread _configtest.o -o _configtest
#6 356.1   success!
#6 356.1   removing: _configtest.c _configtest.o _configtest.o.d _configtest
#6 358.3     C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC
#6 358.3     
#6 358.3     compile options: '-Ibuild/src.linux-x86_64-3.6/numpy/core/src/common -Inumpy/core/include -Ibuild/src.linux-x86_64-3.6/numpy/core/include/numpy -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -Ibuild/src.linux-x86_64-3.6/numpy/core/src/common -Ibuild/src.linux-x86_64-3.6/numpy/core/src/npymath -Ibuild/src.linux-x86_64-3.6/numpy/core/src/common -Ibuild/src.linux-x86_64-3.6/numpy/core/src/npymath -c'
#6 358.3     gcc: build/src.linux-x86_64-3.6/numpy/core/src/npysort/radixsort.c
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'radixsort_bool':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:112:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src:112:5: note: use option -std=c99 or -std=gnu99 to compile your code
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'aradixsort_bool':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:202:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'radixsort_byte':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:112:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'aradixsort_byte':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:202:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'radixsort_ubyte':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:112:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'aradixsort_ubyte':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:202:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'radixsort_short':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:112:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'aradixsort_short':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:202:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'radixsort_ushort':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:112:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'aradixsort_ushort':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:202:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'radixsort_int':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:112:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'aradixsort_int':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:202:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'radixsort_uint':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:112:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'aradixsort_uint':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:202:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'radixsort_long':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:112:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'aradixsort_long':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:202:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'radixsort_ulong':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:112:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'aradixsort_ulong':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:202:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'radixsort_longlong':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:112:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'aradixsort_longlong':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:202:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'radixsort_ulonglong':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:112:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     numpy/core/src/npysort/radixsort.c.src: In function 'aradixsort_ulonglong':
#6 358.3     numpy/core/src/npysort/radixsort.c.src:202:5: error: 'for' loop initial declarations are only allowed in C99 mode
#6 358.3          for (npy_intp i = 1; i < num; i++) {
#6 358.3          ^
#6 358.3     error: Command "gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Ibuild/src.linux-x86_64-3.6/numpy/core/src/common -Inumpy/core/include -Ibuild/src.linux-x86_64-3.6/numpy/core/include/numpy -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -Ibuild/src.linux-x86_64-3.6/numpy/core/src/common -Ibuild/src.linux-x86_64-3.6/numpy/core/src/npymath -Ibuild/src.linux-x86_64-3.6/numpy/core/src/common -Ibuild/src.linux-x86_64-3.6/numpy/core/src/npymath -c build/src.linux-x86_64-3.6/numpy/core/src/npysort/radixsort.c -o build/temp.linux-x86_64-3.6/build/src.linux-x86_64-3.6/numpy/core/src/npysort/radixsort.o -MMD -MF build/temp.linux-x86_64-3.6/build/src.linux-x86_64-3.6/numpy/core/src/npysort/radixsort.o.d" failed with exit status 1
#6 358.3     ----------------------------------------
#6 358.4 ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-poiltggx/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-poiltggx/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-50nzprh1/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.

Numpy/Python version information:

CentOS 7
Default python 3 (3.6)
Default gcc (4.8)

Activity

  1. xkszltl commented on Jul 29, 2019

    @xkszltl
    Author

    Full log attached: numpy.log

  2. eric-wieser commented on Jul 29, 2019

    @eric-wieser
    Member

    Numpy 1.17 needs at least --std=c99.

    I think the decision was to leave passing that flag up to the distributor rather than doing it automatically, but I forget the rationale. @rgommers, were you part of that discussion?

  3. xkszltl commented on Jul 29, 2019

    @xkszltl
    Author

    That sounds terrible if it's not set by default.
    However, it somehow works with Ubuntu 18.04.

    Besides, what's the correct way to pass in compiler flags?
    I tried export CC="gcc --std=c11" and export CFLAGS="--std=c11" but both were ignored.

  4. xkszltl commented on Jul 29, 2019

    @xkszltl
    Author

    Looks like on Ubuntu 18.04, gcc 7.4.0 defaults to gnu11 while on CentOS 7, gcc 4.8.5 defaults to gnu90.

    I run . scl_source enable devtoolset-8 to switch to gcc 8, which uses gnu17 by default.
    But seems all related env vars were ignored by numpy.

  5. xkszltl commented on Jul 29, 2019

    @xkszltl
    Author

    Never mind, numpy can pick up the env correctly.
    There's a sudo before pip install which drops everything.

    But the issue remains: Do we still have the default support for CentOS 7 and its gcc 4.8.5?

  6. paulofbmarcon commented on Jul 31, 2019

    @paulofbmarcon

    I don't know if it's related but I got a similar problem trying to install on Amazon Linux. The following work around helped me:
    update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc48-c99 100 --slave /usr/bin/c89 c89 /usr/bin/gcc48-c89 --slave /usr/bin/c99 c99 /usr/bin/gcc48-c99 --slave /usr/bin/cc cc /usr/bin/gcc48 --slave /usr/bin/gcov gcov /usr/bin/gcov48
    update-alternatives --config gcc

  7. sashkab commented on Jul 31, 2019

    @sashkab

    @xkszltl You should pass --std=c99 in the CFLAGS environmental variable. Following solved build issues for me.

    CFLAGS="--std=c99" python3.6 -mpip wheel "numpy>=1.17,<1.18"
  8. xkszltl commented on Aug 1, 2019

    @xkszltl
    Author

    @sashkab I got it resolved for me by simply using gcc-8. But should --std=c99 be added automatically during build if there's no user setting detected?

  9. antocuni commented on Aug 1, 2019

    @antocuni

    numpy 1.17 is also failing to build on pypy, using a ubuntu 14.04 image: https://travis-ci.org/antocuni/pypy-wheels/jobs/566563056

    it seems to be that numpy should try harder to compile even on older systems, unless you have a good reason not to.

  10. tylerjereddy commented on Aug 5, 2019

    @tylerjereddy
    Contributor

    I think passing in the std flags by default in a compiler and platform-agnostic manner can be tricky & as with many distutils things, hard to test completely.

  11. rgommers commented on Aug 5, 2019

    @rgommers
    Member

    Numpy 1.17 needs at least --std=c99.

    I think the decision was to leave passing that flag up to the distributor rather than doing it automatically, but I forget the rationale. @rgommers, were you part of that discussion?

    That rings a bell. I attempted to fix this before, and concluded there's no good place to insert the fix: #12620 (comment). This attempt has some comment as to what the problem is: rgommers@8aaf4d8

    Taking a step back, the numpy.distutils design problem we run into here is that all flags are (a) taken over from Python distutils, then hardcoded per compiler in numpy/distutils/<compilername>.py. And at the time we do that hardcoding, we don't know what comes from distutils. And inserting flags dynamically later on (what I tried) is equally tricky.

    It's possible I missed a sane way to fix this. Suggestions/PRs welcome.

    IIRC the minimum gcc version that works for building numpy at all is 4.8. For gcc <5.1, users need to add --std=c99.

    This is a duplicate of gh-12766.

  12. added this to the 1.17.1 release milestone on Aug 6, 2019
  13. rgommers commented on Aug 6, 2019

    @rgommers
    Member

    This is biting enough people that I'll mark it for 1.17.1 - should be possible to insert a hacky fix somehow.

  14. 11 remaining items

  15. mattip commented on Nov 11, 2019

    @mattip
    Member

    Fixed in gh-14771, which added both the flag and a CI run with gcc4.8

  16. sh1ng commented on Mar 18, 2020

    @sh1ng

    The issue is reproducible on nvidia/cuda-ppc64le:10.0-cudnn7-devel-centos7

  17. mattip commented on Mar 18, 2020

    @mattip
    Member

    @sh1ng with what version of NumPy? Are you adding the -std=c99 flag?

  18. sh1ng commented on Mar 18, 2020

    @sh1ng

    Without adding the flag

    Numpy version Status
    1.17.0 Fail
    1.17.1 Fail
    1.17.2 Fail
    1.17.3 Fail
    1.17.4 Fail
    1.17.5 Fail
    1.18.0 Success
    1.18.1 Fail
    1.18.2 Fail

    Adding it do solves the issue, but makes the installation process platform dependent.

  19. sh1ng commented on Mar 18, 2020

    @sh1ng

    @mattip the issue occurs when miniconda is used, not a centos python package. Most likely conda's issue.

  20. zoj613 commented on Aug 3, 2021

    @zoj613
    Contributor

    So, which 1.17.* version is this flag automatically set so 1.17.* build can be platform independent? I am having trouble building numpy 1.17.0 on 3.8 and 3.9 for CI jobs.

  21. sashkab commented on Aug 3, 2021

    @sashkab

    @zoj613 did you try my suggestion above with python 3.8? It worked for me for both 1.16.x and 1.17.x on RHEL/Centos 7.

  22. charris commented on Aug 3, 2021

    @charris
    Member

    @zoj613 It isn't automatically set in any 1.17 version, you need 1.18.0+ for that.

  23. zoj613 commented on Aug 3, 2021

    @zoj613
    Contributor

    Ah, Okay. @sashkab 's suggestion seems to work fine on 1.17.3

  24. zevaverbach commented on Jun 8, 2022

    @zevaverbach

    @xkszltl You should pass --std=c99 in the CFLAGS environmental variable. Following solved build issues for me.

    CFLAGS="--std=c99" python3.6 -mpip wheel "numpy>=1.17,<1.18"

    just to build on this, for anyone that's lost for a next step: pip install <the_wheel_the_above_step_creates>.whl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions