BUG: Test failures with 1.26.0rc1 in conda-forge #24660
Copy link
Copy link
Closed
Labels
Milestone
Description
Describe the issue:
While our CI for 1.26.0b1 was all green, for rc1 we have two issues in conda-forge/numpy-feedstock#297:
linux-aarch64 has two test failures that don't just seem like minor tolerance violations:
=========================== short test summary info ============================
FAILED linalg/tests/test_linalg.py::TestCholesky::test_basic_property[float32-shape3] - AssertionError:
Not equal to tolerance rtol=1e-07, atol=0.00298023
[...]
Mismatched elements: 1617 / 2500 (64.7%)
Max absolute difference: 89.25262
Max relative difference: 2200.9631
x: array([[ 4.406959e+01, -1.291737e+00, -6.148944e-01, ..., -9.883090e+00,
-2.005572e+00, 7.965497e+00],
[-1.291737e+00, -7.090253e+00, 1.104174e+00, ..., -9.404308e-03,...
y: array([[ 44.06959 , -1.291737, -0.614894, ..., -1.707806, -0.06027 ,
3.596826],
[ -1.291737, 50.031647, -0.936252, ..., -9.246351, -12.977635,...
FAILED linalg/tests/test_linalg.py::TestCholesky::test_basic_property[float32-shape4] - AssertionError:
Not equal to tolerance rtol=1e-07, atol=0.000178814
[...]
Mismatched elements: 253 / 300 (84.3%)
Max absolute difference: 20.29966
Max relative difference: 237.33156
x: array([[[10.734334, -4.876828, -3.155457, -4.599753, 2.103036,
-1.243519, 3.030686, -4.149522, 0. , 0. ],
[-4.876828, 2.244246, 1.433589, 8.771067, 1.423618,...
y: array([[[10.734333, -4.876828, -3.155457, -4.599753, 2.103036,
-1.243519, 3.030686, -4.149522, -2.962044, 0.036255],
[-4.876828, 8.896948, 3.812659, -0.037113, -1.047684,...
2 failed, 34361 passed, 1117 skipped, 33 xfailed, 4 xpassed, 31 warnings in 2741.47s (0:45:41)
and pypy on windows, which seems to have some issues with cython resp. f2py:
=========================== short test summary info ===========================
FAILED random/tests/test_extending.py::test_cython - subprocess.CalledProcess...
FAILED tests/test_scripts.py::test_pep338 - subprocess.CalledProcessError: Co...
ERROR core/tests/test_mem_policy.py::test_set_policy - RuntimeError: could no...
ERROR core/tests/test_mem_policy.py::test_default_policy_singleton - RuntimeE...
ERROR core/tests/test_mem_policy.py::test_policy_propagation - RuntimeError: ...
ERROR core/tests/test_mem_policy.py::test_context_locality - RuntimeError: co...
ERROR core/tests/test_mem_policy.py::test_thread_locality - RuntimeError: cou...
ERROR core/tests/test_mem_policy.py::test_owner_is_base - RuntimeError: could...
2 failed, 38142 passed, 671 skipped, 36 xfailed, 26 xpassed, 23 warnings, 6 errors in 1984.29s (0:33:04)
Reproduce the code example:
Run tests in conda-forge CIError message:
See aboveRuntime information:
Test env aarch:
_openmp_mutex: 4.5-2_gnu conda-forge
_sysroot_linux-aarch64_curr_repodata_hack: 4-h57d6b7b_13 conda-forge
attrs: 23.1.0-pyh71513ae_1 conda-forge
backports.zoneinfo: 0.2.1-py311hec3470c_7 conda-forge
binutils_impl_linux-aarch64: 2.40-h870a726_0 conda-forge
binutils_linux-aarch64: 2.40-h94bbfa1_1 conda-forge
bzip2: 1.0.8-hf897c2e_4 conda-forge
ca-certificates: 2023.7.22-hcefe29a_0 conda-forge
click: 8.1.7-unix_pyh707e725_0 conda-forge
colorama: 0.4.6-pyhd8ed1ab_0 conda-forge
cython: 3.0.2-py311h8715677_0 conda-forge
exceptiongroup: 1.1.3-pyhd8ed1ab_0 conda-forge
execnet: 2.0.2-pyhd8ed1ab_0 conda-forge
gcc_impl_linux-aarch64: 12.3.0-hcde2664_0 conda-forge
gcc_linux-aarch64: 12.3.0-h464a8f7_1 conda-forge
gfortran_impl_linux-aarch64: 12.3.0-hb7244be_0 conda-forge
gfortran_linux-aarch64: 12.3.0-h1993883_1 conda-forge
gxx_impl_linux-aarch64: 12.3.0-hcde2664_0 conda-forge
gxx_linux-aarch64: 12.3.0-h21accf6_1 conda-forge
hypothesis: 6.84.2-pyha770c72_0 conda-forge
iniconfig: 2.0.0-pyhd8ed1ab_0 conda-forge
kernel-headers_linux-aarch64: 4.18.0-h5b4a56d_13 conda-forge
ld_impl_linux-aarch64: 2.40-h2d8c526_0 conda-forge
libblas: 3.9.0-18_linuxaarch64_openblas conda-forge
libcblas: 3.9.0-18_linuxaarch64_openblas conda-forge
libexpat: 2.5.0-hd600fc2_1 conda-forge
libffi: 3.4.2-h3557bc0_5 conda-forge
libgcc-devel_linux-aarch64: 12.3.0-h8b5ab12_0 conda-forge
libgcc-ng: 13.1.0-h2b4548d_0 conda-forge
libgfortran-ng: 13.1.0-he9431aa_0 conda-forge
libgfortran5: 13.1.0-h24e4805_0 conda-forge
libgomp: 13.1.0-h2b4548d_0 conda-forge
liblapack: 3.9.0-18_linuxaarch64_openblas conda-forge
libnsl: 2.0.0-hf897c2e_0 conda-forge
libopenblas: 0.3.24-pthreads_h5a5ec62_0 conda-forge
libsanitizer: 12.3.0-h8ebda82_0 conda-forge
libsqlite: 3.43.0-h194ca79_0 conda-forge
libstdcxx-devel_linux-aarch64: 12.3.0-h8b5ab12_0 conda-forge
libstdcxx-ng: 13.1.0-h452befe_0 conda-forge
libuuid: 2.38.1-hb4cce97_0 conda-forge
libzlib: 1.2.13-h31becfc_5 conda-forge
meson: 1.2.1-pyhd8ed1ab_0 conda-forge
ncurses: 6.4-h2e1726e_0 conda-forge
ninja: 1.11.1-hdd96247_0 conda-forge
numpy: 1.26.0rc1-py311h69ead2a_0 local
openssl: 3.1.2-h31becfc_0 conda-forge
packaging: 23.1-pyhd8ed1ab_0 conda-forge
pluggy: 1.3.0-pyhd8ed1ab_0 conda-forge
pytest: 7.4.1-pyhd8ed1ab_0 conda-forge
pytest-timeout: 2.1.0-pyhd8ed1ab_0 conda-forge
pytest-xdist: 3.3.1-pyhd8ed1ab_0 conda-forge
python: 3.11.5-h43d1f9e_0_cpython conda-forge
python_abi: 3.11-3_cp311 conda-forge
pytz: 2023.3.post1-pyhd8ed1ab_0 conda-forge
readline: 8.2-h8fc344f_1 conda-forge
setuptools: 59.8.0-py311hec3470c_1 conda-forge
sortedcontainers: 2.4.0-pyhd8ed1ab_0 conda-forge
sysroot_linux-aarch64: 2.17-h5b4a56d_13 conda-forge
tk: 8.6.12-hd8af866_0 conda-forge
tomli: 2.0.1-pyhd8ed1ab_0 conda-forge
typing_extensions: 4.7.1-pyha770c72_0 conda-forge
tzdata: 2023c-h71feb2d_0 conda-forge
xz: 5.2.6-h9cdd2b7_0 conda-forge
Test env pypy on windows
attrs: 23.1.0-pyh71513ae_1 conda-forge
backports.zoneinfo: 0.2.1-py39h0d475fb_7 conda-forge
bzip2: 1.0.8-h8ffe710_4 conda-forge
ca-certificates: 2023.7.22-h56e8100_0 conda-forge
click: 8.1.7-win_pyh7428d3b_0 conda-forge
colorama: 0.4.6-pyhd8ed1ab_0 conda-forge
cython: 3.0.2-py39h3665ca7_0 conda-forge
exceptiongroup: 1.1.3-pyhd8ed1ab_0 conda-forge
execnet: 2.0.2-pyhd8ed1ab_0 conda-forge
expat: 2.5.0-h63175ca_1 conda-forge
hypothesis: 6.84.2-pyha770c72_0 conda-forge
iniconfig: 2.0.0-pyhd8ed1ab_0 conda-forge
intel-openmp: 2023.2.0-h57928b3_49496 conda-forge
libblas: 3.9.0-18_win64_mkl conda-forge
libcblas: 3.9.0-18_win64_mkl conda-forge
libexpat: 2.5.0-h63175ca_1 conda-forge
libffi: 3.4.2-h8ffe710_5 conda-forge
libhwloc: 2.9.2-default_haede6df_1009 conda-forge
libiconv: 1.17-h8ffe710_0 conda-forge
liblapack: 3.9.0-18_win64_mkl conda-forge
libsqlite: 3.43.0-hcfcfb64_0 conda-forge
libxml2: 2.11.5-hc3477c8_1 conda-forge
libzlib: 1.2.13-hcfcfb64_5 conda-forge
meson: 1.2.1-pyhd8ed1ab_0 conda-forge
mkl: 2022.1.0-h6a75c08_874 conda-forge
ninja: 1.11.1-h91493d7_0 conda-forge
numpy: 1.26.0rc1-py39h821adfb_0 local
openssl: 3.1.2-hcfcfb64_0 conda-forge
packaging: 23.1-pyhd8ed1ab_0 conda-forge
pluggy: 1.3.0-pyhd8ed1ab_0 conda-forge
pthreads-win32: 2.9.1-hfa6e2cd_3 conda-forge
pypy3.9: 7.3.12-h994e1e7_4 conda-forge
pytest: 7.4.1-pyhd8ed1ab_0 conda-forge
pytest-timeout: 2.1.0-pyhd8ed1ab_0 conda-forge
pytest-xdist: 3.3.1-pyhd8ed1ab_0 conda-forge
python: 3.9.17-0_73_pypy conda-forge
python_abi: 3.9-3_pypy39_pp73 conda-forge
pytz: 2023.3.post1-pyhd8ed1ab_0 conda-forge
setuptools: 59.8.0-py39h0d475fb_1 conda-forge
sortedcontainers: 2.4.0-pyhd8ed1ab_0 conda-forge
sqlite: 3.43.0-hcfcfb64_0 conda-forge
tbb: 2021.10.0-h91493d7_0 conda-forge
tk: 8.6.12-h8ffe710_0 conda-forge
tomli: 2.0.1-pyhd8ed1ab_0 conda-forge
typing_extensions: 4.7.1-pyha770c72_0 conda-forge
tzdata: 2023c-h71feb2d_0 conda-forge
ucrt: 10.0.22621.0-h57928b3_0 conda-forge
vc: 14.3-h64f974e_17 conda-forge
vc14_runtime: 14.36.32532-hfdfe4a8_17 conda-forge
vs2015_runtime: 14.36.32532-h05e6639_17 conda-forge
xz: 5.2.6-h8d14728_0 conda-forge
zlib: 1.2.13-hcfcfb64_5 conda-forge
Context for the issue:
No response
Reactions are currently unavailable
Activity
h-vetinari commentedon Sep 7, 2023
Full error log linux-aarch
Full error log pypy on windows
charris commentedon Sep 7, 2023
Which openblas version are you using?
Which cython version are you using?
Did you build using meson or distutils?
The cholesky error is most likely due to blas.
The only changes in
numpy/randomwere changing the cython binding directive (#24606) and a slight annotation improvement. The wheels we built did not error.h-vetinari commentedon Sep 7, 2023
I put the details in the issue text (under
Runtime information:); it's openblas 0.3.24 and Cython 3.0.2.Meson on all platforms.
Yeah, 0.3.24 came out not too long ago, I can retest with 0.3.23.
mattip commentedon Sep 7, 2023
I think the previous 1.26.0b1 may have been using lapack-lite for nmpy.linalg by mistake instead of OpenBLAS
h-vetinari commentedon Sep 7, 2023
Can confirm that aarch passes with openblas 0.3.23.
@martin-frbg, this looks like a regression in 0.3.24. Do you want me to open an issue for OpenBLAS or perhaps you have a possible culprit in mind already based on the error in the cholesky test?
martin-frbg commentedon Sep 7, 2023
Which flavor of aarch64 is this ? The only very recent change that made it into 0.3.24 was the ROTG rewrite to use the new "safe" scaling algorithm from the Reference BLAS, most everything else was specific to NeoverseV1.
h-vetinari commentedon Sep 7, 2023
Since this is running in emulation through QEMU, I'm guessing it's the "most vanilla possible"? Numpy has some CPU feature detection on aarch as well, but doesn't find any CPU features (printed before the test suite run).
martin-frbg commentedon Sep 7, 2023
That would be the ARMV8 target, no changes there and nothing on the obvious Cholesky codepath (potrf &friends)
mattip commentedon Sep 7, 2023
I wonder what the difference is between this aarch64 test run and the one in the github action on the numpy/numpy repo
charris commentedon Sep 8, 2023
What pypy version?
seiko2plus commentedon Sep 8, 2023
According to the build log:
The compile-time test for NEON_VFPV4 located at https://github.com/numpy/numpy/blob/main/numpy/distutils/checks/cpu_neon_vfpv4.c seems to have failed. Is there a way we can access the meson build log to pinpoint the problem? Perhaps by using something like
cat build/meson-logs/meson-log.txt?h-vetinari commentedon Sep 8, 2023
Also in the OP already: pypy 7.3.12 (for python 3.9, as numpy doesn't support 3.8 anymore, and conda-forge doesn't build pypy for 3.10 yet)
h-vetinari commentedon Sep 8, 2023
I think these are always going to fail because we're cross-compiling on aarch. It sounds conceptually similar to #24414, which where we have to set an explicit value for cross-compilation to pass for osx-arm.
charris commentedon Sep 8, 2023
The hash in the OP didn't agree with the latest pypy release, that is why I asked.
8 remaining items
BLD: fix issue with compiler selection during cross compilation
rgommers commentedon Sep 13, 2023
gh-24698 should close the important part of this issue, which is the Linux aarch64 accuracy issues.
The PyPy on Windows + memory allocator / Cython / f2py are all the same issue it looks like:
sys.stdoutnot having an "encoding" attribute:I'm not sure why
sys.stdoutis weird under PyPy on Windows there, but it's a test issue only and shouldn't impact anything of interest. We have a PyPy on Windows CI job, but it's running the default (fast) test suite rather than the full test suite. @h-vetinari it should be fine to mark these as broken to get the conda-forge CI to pass I think. We may want to open a follow-up issue here, but it's too low-prio to look at now imho.BLD: fix issue with compiler selection during cross compilation
h-vetinari commentedon Sep 13, 2023
Thanks a lot @seiko2plus!
Is this going to be upstreamed into meson? Because if I mark the openblas 0.3.24 builds unbroken again on aarch, this would also rebreak the CI on the scipy-feedstock (which, being built by meson, is probably affected by the same issue).
In other words, we'd at least need to patch this into the meson-feedstock, but I'd prefer to do just a backport of an upstream PR, rather than original patches.
h-vetinari commentedon Sep 13, 2023
Agreed, issues with pypy on windows would not have kept us from releasing (on the conda-forge side at least)
rgommers commentedon Sep 13, 2023
This bug is in code that is not present in Meson and not relevant for SciPy because it doesn't use NEON SIMD.
h-vetinari commentedon Sep 13, 2023
Interesting... That means we're still lacking an understanding of what's causing scipy/scipy#19210, and I cannot mark openblas 0.3.24 unbroken again.
In any case, nothing that should be blocking 1.26
seiko2plus commentedon Sep 14, 2023
The issues seem to stem from the compiler optimization being disabled by setting the optimize level to 0. Upon inspecting the provided meson log, I noticed that -O0 is specified after both -O3 and -O2, which is unusual.
Check the following command line:
seiko2plus commentedon Sep 14, 2023
It seems that the Meson build reverses the order of the CFLAGS for somehow and this may explains why we don't observe this issue in the distutils build.
rgommers commentedon Sep 14, 2023
Conda manages to make things a little messy with a ton of build flags, but I think we can unravel this. We have the following components of the puzzle for a C target:
default_optionsin the top-levelmeson.buildfile:-std=c99(other relevant flags there get overridden by meson-python, see (2):)meson-pythonadds when building a wheel:-Dbuildtype=release,-Db_ndebug=if_release`c_argsset for a particular build target in the relevantmeson.buildfileCFLAGSandLDFLAGSset in the environment (comes from compiler activation):It's unclear to me where
-O0comes from, or-Wl,-O2.To get the compile and link args for a target, you can use
meson introspect build --targets -i > tmp.json. Example output in anumpy-devconda env:Details
Comparing that to a non-SIMD compiler feature check and a SIMD feature check from
meson-log.txton macOS arm64:Details
Those do both contain
-O0, so I think it's coming from Meson's compiler feature check support. Which I guess makes sense forcc.has_headerand the like. But not so much for SIMD checks perhaps.rgommers commentedon Sep 14, 2023
@seiko2plus are those actual problems for feature detection? It's working fine in all CI and locally, with
-O0included.seiko2plus commentedon Sep 14, 2023
Thanks for clarifying. I was under the impression -O0 was applied to all sources, as I couldn't access the ninja debug log via CI.
The presence of -O0 isn't a problem for feature detection. In fact, it's beneficial to include
-O0to prevent the optimization of any instructions pertinent to the test, allowing for direct assessment against the assembler.rgommers commentedon Sep 14, 2023
Great, thanks for confirming. Then I think we're all good here.
BLD: fix issue with compiler selection during cross compilation