Test suite failure on non-AVX512 due to glibc (version-dependent) bug on f64 underflow

@seiko2plus

While trying to build 1.21.0rc1 for conda-forge, a new bug appeared, for agents without AVX512 support.

Test suite failure for TestSpecialFloats.test_exp_values

=================================== FAILURES ===================================
______________________ TestSpecialFloats.test_exp_values _______________________

self = <numpy.core.tests.test_umath.TestSpecialFloats object at 0x7f1a528bb820>

    def test_exp_values(self):
        x = [np.nan,  np.nan, np.inf, 0.]
        y = [np.nan, -np.nan, np.inf, -np.inf]
        for dt in ['f', 'd', 'g']:
            xf = np.array(x, dtype=dt)
            yf = np.array(y, dtype=dt)
            assert_equal(np.exp(yf), xf)
    
        with np.errstate(over='raise'):
            assert_raises(FloatingPointError, np.exp, np.float32(100.))
            assert_raises(FloatingPointError, np.exp, np.float32(1E19))
            assert_raises(FloatingPointError, np.exp, np.float64(800.))
            assert_raises(FloatingPointError, np.exp, np.float64(1E19))
    
        with np.errstate(under='raise'):
            assert_raises(FloatingPointError, np.exp, np.float32(-1000.))
            assert_raises(FloatingPointError, np.exp, np.float32(-1E19))
>           assert_raises(FloatingPointError, np.exp, np.float64(-1000.))

../[...]/lib/python3.9/site-packages/numpy/core/tests/test_umath.py:998: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../[...]/lib/python3.9/unittest/case.py:733: in assertRaises
    return context.handle('assertRaises', args, kwargs)
../[...]/lib/python3.9/unittest/case.py:201: in handle
    callable_obj(*args, **kwargs)
../[...]/lib/python3.9/unittest/case.py:223: in __exit__
    self._raiseFailure("{} not raised by {}".format(exc_name,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <unittest.case._AssertRaisesContext object at 0x7f1a528bba90>
standardMsg = 'FloatingPointError not raised by exp'

    def _raiseFailure(self, standardMsg):
        msg = self.test_case._formatMessage(self.msg, standardMsg)
E       AssertionError: FloatingPointError not raised by exp

../[...]/lib/python3.9/unittest/case.py:163: AssertionError
=============================== warnings summary ===============================

@seiko2plus commented:

It makes sense now the error disappears when the machine has AVX512 support, it seems we are dealing with compiler/libc that doesn't respect underflow error since the current exp/f64 SIMD kernel only optimizes CPUs with AVX512 Skylake features.

[...]

However, Could you try the following patch?

Details

index 41e0bf37b..89798f74a 100644
--- a/numpy/core/src/umath/loops_exponent_log.dispatch.c.src
+++ b/numpy/core/src/umath/loops_exponent_log.dispatch.c.src
@@ -1248,6 +1248,7 @@ NPY_NO_EXPORT void NPY_CPU_DISPATCH_CURFX(FLOAT_@func@)
 /**begin repeat
  * #func = exp, log#
  * #scalar = npy_exp, npy_log#
+ * #underflow = 1, 0#
  */
 NPY_NO_EXPORT void NPY_CPU_DISPATCH_CURFX(DOUBLE_@func@)
 (char **args, npy_intp const *dimensions, npy_intp const *steps, void *NPY_UNUSED(data))
@@ -1260,6 +1261,13 @@ NPY_NO_EXPORT void NPY_CPU_DISPATCH_CURFX(DOUBLE_@func@)
 #endif
     UNARY_LOOP {
         const npy_double in1 = *(npy_double *)ip1;
+    #if @underflow@
+        if (NPY_UNLIKELY(!npy_isnan(in1) && in1 < -0x1.74910d52d3053p+9)) {
+            *(npy_double *)op1 = 0;
+            npy_set_floatstatus_underflow();
+            continue;
+        }
+    #endif
         *(npy_double *)op1 = @scalar@(in1);
     }
 }

if the patch passes the test, then we will need to determine which versions of glibc are affected by this bug.

And indeed, the test passes with that patch, which is why I'm opening this issue.

Sidenote: based on the mention of (g)libc, this might be somewhat similar to #15179? The corresponding test is currently being skipped in conda-forge, but does pass with the following patch (suggested by @r-devulap):

Details

diff --git a/numpy/core/tests/test_umath.py b/numpy/core/tests/test_umath.py
index 9d1b13b53..faa2ca8f0 100644
--- a/numpy/core/tests/test_umath.py
+++ b/numpy/core/tests/test_umath.py
@@ -1188,8 +1188,6 @@ def test_sincos_float32(self):
         M = np.int_(N/20)
         index = np.random.randint(low=0, high=N, size=M)
         x_f32 = np.float32(np.random.uniform(low=-100.,high=100.,size=N))
-        # test coverage for elements > 117435.992f for which glibc is used
-        x_f32[index] = np.float32(10E+10*np.random.rand(M))
         x_f64 = np.float64(x_f32)
         assert_array_max_ulp(np.sin(x_f32), np.float32(np.sin(x_f64)), maxulp=2)
         assert_array_max_ulp(np.cos(x_f32), np.float32(np.cos(x_f64)), maxulp=2

@h-vetinari

Hi @h-vetinari, sorry for not getting back to this. As @seiko2plus mentioned, this is a bug in the glibc library. Could you tell us OS and the glibc version on the machine this fails? As a workaround, we could patch NumPy with what @seiko2plus has suggested, if that is acceptable to NumPy maintainers.

@r-devulap

Hey @r-devulap, no problem, we've all got a lot of things going on. 🙃

The failing runs all had glibc = 2.12 (corresponding to CentOS 6). If it helps, I could see if the error disappears for 2.17 (CentOS 7)

Cf. similar discussion (about glibc verison) in #15179.

oh yikes, 2.12 is a decade old! Perhaps, as you suggested, an easier solution is just to skip this test for glibc < 2.17.

@r-devulap

I could see if the error disappears for 2.17

We could put out wheels for 2_17 (manylinux2014). We currently do that for aarch64. @r-devulap I was hoping for a fix before the 1.21.0 release.

oh yikes, 2.12 is a decade old!

Yeah, this is a similar situation as with multilinux - conda-forge tries to build for the lowest common denominator, which currently is CentOS 6, but there slowly are some packages requiring CentOS 7.

Perhaps, as you suggested, an easier solution is just to skip this test for glibc < 2.17.

I mean, I don't see this test failure (of not raising an error) to be super critical. But I've just pushed a commit to conda-forge/numpy-feedstock#235 to see if the failure is gone with glibc 2.17. However, I expect the bump on conda-forge side to 2.17 to not be merged, because that would force all packages that depend on numpy to also require 2.17.

Certainly, if the above patch is not too painful for numpy, I think that would be the best solution.

Quick update: with glibc 2.17, the bug disappears

Yeah, this is a similar situation as with multilinux - conda-forge tries to build for the lowest common denominator, which currently is CentOS 6, but there slowly are some packages requiring CentOS 7.
....
However, I expect the bump on conda-forge side to 2.17 to not be merged, because that would force all packages that depend on numpy to also require 2.17.

CentOS 6 is >6 months past EOL. Official EOL really should be the last time to drop a constraint. The patch for working around this problem seems fine to accept, but exercises like these are a massive waste of time. For manylinux we're also fine to move to glibc 2.17 (= manylinux2014), for the same reason. What's the problem here exactly?

What's the problem here exactly?

If we want to continue to support manylinux2010, we need to adopt the patch above with a #ifdef for glibc < 2.17. If not, we need to be absolutely sure pip 19.3+ released Oct 2019 is the oldest version in use, older pips will download a source package and try to compile.

If not, we need to be absolutely sure pip 19.3+ released Oct 2019 is the oldest version in use, older pips will download a source package and try to compile.

It'd be nice if we could raise an informative error from setup.py or specify a constraint on pip version - but I don't think either is possible in a reliable way. I had a look for a feature request on the pip issue tracker, but couldn't find anything.

People use 5-6 year old Python installs on clusters sometimes, so that'll also come with old Pip versions probably. That's not a great reason to continue to support an OS that's end-of-life for several more years though. It's maybe something we should bring up with the packaging crowd.

I think all the logic whether to choose a sdist or a wheel is part of pip. But maybe we could encourage PyPI to modify that for pre-20.0 pip?

I opened an issue in pypi/warehouse#9640

@seiko2plus

Over on the warehouse issue, @di proposed this idea:

It seems like the root of the problem is:

users are using an old version of pip
users are trying to install an unpinned version of numpy

Therefore, if numpy's setup.py was able to detect the installing pip version and, for these versions of pip emit an error failing installation and telling the user to ether:

upgrade their pip
pin to some specific last-known version of numpy compatible with the pip version in use
(ideally, do both)

Whichever action the user takes would resolve the issue for them.
This would effectively mean that anyone installing an unpinned numpy with an old version of pip would get a deployment/installation failure, but I think that's an accepted risk when not pinning versions.

I don't know if we have the information to make this work, but it is intriguing.

Of course it doesn't solve the immediate problem in this issue of getting @seiko2plus 's patch in and working for the upcoming releasel

@rgommers

... and @rgommers said the same thing here earlier, so 🤷

Therefore, if numpy's setup.py was able to detect the installing pip version and, for these versions of pip emit an error failing installation

Yes, that would solve it indeed, but I suspect it's not possible. In setup.py we could be in any of these situations:

Invoked from pip in an isolated env - pip itself is not installed at all in that env.
Invoked from pip in a non-isolated env - pip version installed could be checked, but it's hard to be sure pip is doing the installing here.
Invoked by the end user, e.g. python setup.py install
Invoked by another installer, e.g. https://github.com/pradyunsg/installer

pip does have a fairly specific way of calling setup.py, using --single-version-externally-managed, but AFAIK that's not unique to pip and best-practice for other installation tools as well. Also, the exact invocation isn't guaranteed to be stable over pip releases. Looking at sys.argv and having some heuristics for "this looks like it came from pip" is probably the best we can do.

Please see #19209. Let me know if that is okay.

Navigation Menu

Repository navigation

Test suite failure on non-AVX512 due to glibc (version-dependent) bug on f64 underflow #19192

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Test suite failure on non-AVX512 due to glibc (version-dependent) bug on f64 underflow #19192

Description

Activity

r-devulap commented on Jun 8, 2021

h-vetinari commented on Jun 8, 2021

r-devulap commented on Jun 8, 2021

charris commented on Jun 8, 2021

h-vetinari commented on Jun 8, 2021

h-vetinari commented on Jun 8, 2021

rgommers commented on Jun 9, 2021

mattip commented on Jun 9, 2021

rgommers commented on Jun 9, 2021

mattip commented on Jun 9, 2021

mattip commented on Jun 9, 2021

mattip commented on Jun 9, 2021

mattip commented on Jun 9, 2021

rgommers commented on Jun 9, 2021

r-devulap commented on Jun 9, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions