Repository navigation
CI: Test Linux (nightly OpenBLAS) is failing #26824
Description
See https://github.com/numpy/numpy/actions/runs/9732669831/job/26858622819. Seems to have started yesterday (6/30/2024)
See https://github.com/numpy/numpy/actions/runs/9732669831/job/26858622819. Seems to have started yesterday (6/30/2024)
Activity
mattip commentedon Jul 2, 2024
This run is using scipy-openblas32-0.3.27.233 (aka OpenBLAS commit
c33bc849a, where last week's scipy-openblas32-0.3.27.208 (aka OpenBLAS commit1ba1b9c35) did not have this problem. @martin-frbg any thoughts?martin-frbg commentedon Jul 2, 2024
Probably caused by that commit enforcing
0*INF=NANto satisfy OpenMathLib/OpenBLAS#4728 . I need to doublecheck the claim made there that the reference BLAS really does this, as that change appears to introduce a significant number of LAPACK test failures on x86_64martin-frbg commentedon Jul 7, 2024
Still haven't got a handle on this, sorry - I have convinced myself that the changes from OpenMathLib/OpenBLAS#4728 are correct, but there appears to be a problem with NAN propagation elsewhere in the code (and unrelated to the unmerged PR about NAN sort order in IMAX/IMIN, OpenMathLib/OpenBLAS#4191 ). Unfortunately I still have a hard time matching up Numpy tests with the BLAS functions they call.
The LAPACK test failures appear to be a different issue, working around them does not change the numpy test results
mattip commentedon Jul 8, 2024
Thanks for looking into it. Maybe we should mark the tests as xfail with that version of scipy-openblas32 to clear CI?
martin-frbg commentedon Jul 8, 2024
probably, or rollback scipy-openblas32 - I have no idea what makes more sense in your context ?
I'll try to fix whatever this is ASAP as it is blocking the already-delayed 0.3.28, but I'd rather not revert the (valid) PR that uncovered this problem
mattip commentedon Jul 8, 2024
We are using the "nightly sicpy-openblas" builds specifically to smoke out problems exactly like this, so it is on NumPy to mitigate the CI failure when we discover one. Sorry I cannot help right now simplifying the test to a clear OpenBLAS call.
ngoldbaum commentedon Jul 19, 2024
I opened #26991 to temporarily allow the scipy-openblas nightly tests to fail without failing all of the CI. Just the test step in the CI job, and hopefully only temporarily.
CI: Re-enable nightly OpenBLAS test runs
CI: Re-enable nightly OpenBLAS test runs