Repository navigation
polyfit and eig regression tests fail after Windows 10 update to 2004 #16744
Description
Activity
Edit: You are obviously using pip.I have also had a strange result on Windows AMD64 in the recent past with linear algebra libraries and eigenvalue decompositions (in the context of running test for statsmodels).
If you have time, try using 32 bit Python and pip and see if you get the same issues? I couldn't see them on 32-bit windows but they were repeatable on 64-bit windows.
If I use conda, which ships MKL, I don't have the errors.
Edit: I also see them when using NumPy 1.18.5 on Windows AMD64.
tests fail after latest Windows 10 update
Were the tests failing before the update?
1909 wasn't causing anything as well. It only started happening with 2004.
I'm not 100% convinced it is 2004 or something after. I think 2004 worked.
FWIW I can't reproduce these crashes on CI (Azure or appveyor) but I do it is locally on 2 machines that are both AMD64 and update on 2004.
Have either of you tried to see if you get them on 32-bit Python?
Seems there have been a number of problems reported against the 2004 update. Maybe this should be reported also?
I just ran the following on fresh install of 1909 and 2004:
pip install numpy scipy pandas pytest cython git clone https://github.com/statsmodels/statsmodels.git cd statsmodels pip install -e . --no-bulid-isolation pytest statsmodelsOn 1909 no failures. On 2004 30 failures all related to linear algebra functions.
When I run tests on 2004 in a debugger, I notice that the first call to a function often returns in incorrect result, but calling again produces the correct result (which remains correct if repeatedly called). Not sure if this is useful information as to guessing a cause.
Do earlier versions of NumPy also have problems? I assume you are running 1.19.0.
- changed the title
[-]polyfit and eig regretion tests fail after latest Windows 10 update[/-][+]polyfit and eig regression tests fail after latest Windows 10 update[/+]on Jul 8, 2020 Using pip + 1.18.4, and scipy 1.4.1, I get the same set of errors.
These are really common:
ERROR statsmodels/graphics/tests/test_gofplots.py::TestProbPlotLongely::test_ppplot - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/graphics/tests/test_gofplots.py::TestProbPlotLongely::test_qqplot_other_array - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/graphics/tests/test_gofplots.py::TestProbPlotLongely::test_probplot - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/graphics/tests/test_regressionplots.py::TestPlot::test_plot_leverage_resid2 - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/regression/tests/test_regression.py::TestOLS::test_params - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/regression/tests/test_regression.py::TestOLS::test_scale - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/regression/tests/test_regression.py::TestOLS::test_ess - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/regression/tests/test_regression.py::TestOLS::test_mse_total - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/regression/tests/test_regression.py::TestOLS::test_bic - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/regression/tests/test_regression.py::TestOLS::test_norm_resids - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/regression/tests/test_regression.py::TestOLS::test_HC2_errors - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/regression/tests/test_regression.py::TestOLS::test_missing - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/regression/tests/test_regression.py::TestOLS::test_norm_resid_zero_variance - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/tsa/tests/test_stattools.py::TestADFConstant::test_teststat - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/tsa/tests/test_stattools.py::TestADFConstant::test_pvalue - numpy.linalg.LinAlgError: SVD did not converge ERROR statsmodels/tsa/tests/test_stattools.py::TestADFConstant::test_critvalues - numpy.linalg.LinAlgError: SVD did not convergeWhen I run using 1.18.5 + MKL I get no errors. It is hard to say whether this is likely an OpenBLAS bug or a Windows bug. Probably the latter, but it will be really hard to get to and diagnosing is beyond my capabilities for low-level debugging.
On the same physical machine, when I run in Ubuntu using pip packages I don't see any errors.
Reacted by Sungwook Yeom and Peter KovalThis is the strangest behavior. Fails on first call, works on second and subsequent. One other hard to understand behavior is that if I run the failing test in isolation than I don't see the error.
174 remaining items
- added a commit that references this issue
on Nov 29, 2020 - added a commit that references this issue
on Nov 29, 2020 There is a new comment from Steve Wishnousky from 5 Dez at fmod(), after an update to windows 2004, is causing a strange interaction with other code:
An Insider Preview Windows build with the fix is now available. Build 20270 is now in the Dev Channel: announcing-windows-10-insider-preview-build-20270. If you are affected by this issue and want to test against the fix that will come via Windows Update, you can use this build. This is an insider build with other changes included, of course, so it is not entirely representative of the version that will be serviced, but it can be used to test for specific cases where the bug used to reproduce. We are still on schedule for servicing Windows build 19041 (the current release version) at the end of January 2021.
Indeed, after updating to the latest Win 10 Dev Channel track (incl "Windows 10 Insider Preview 20270.1 (fe_release)"),
numpy==1.19.4no longer triggers the sanity check and works as expected.Today this fix is available only in Dev Channel track, not yet in Beta nor Release Preview.
Hopefully it will be downstream next month and this can be closed.
I just updated from Windows 2004 to Windows 20H2 (OS Build: 19042.685), and unexpectedly, the import error under
1.19.4(on the sanity check) that I had just encountered before the upgrade is gone now - I also double-checked the reproducing example manually, and it's not segfaulting.>>> import numpy as np >>> a = np.arange(13 * 13, dtype=np.float64) >>> a.shape = (13, 13) >>> a = a % 17 >>> va, ve = np.linalg.eig(a) >>> va array([ 1.03221168e+02 +0.j , -1.91843603e+01 +0.j , 1.82126812e+01 +0.j , -6.04004526e-01+15.84422474j, -6.04004526e-01-15.84422474j, -1.13692929e+01 +0.j , -6.57612485e-01+10.41755503j, -6.57612485e-01-10.41755503j, 1.06011014e+01 +0.j , 7.80732773e+00 +0.j , -7.65390898e-01 +0.j , 2.87796761e-15 +0.j , -1.26447204e-15 +0.j ])
I then rechecked my
C:\Windows\System32\ucrtbase.dll, and even more strangely, the explorer mouse-over tells me this is10.0.19041.546from10/16/2020(i.e. definitely older than some comments here, but then I have to admit I don't know what it was before the upgrade).Not sure what's happening, but thought I'd let people know... :)
PS.
>>> import sys, numpy; print(numpy.__version__, sys.version) 1.19.4 3.8.6 | packaged by conda-forge | (default, Dec 22 2020, 09:52:49) [MSC v.1916 64 bit (AMD64)]
Reacted by Steve DowerThe date on the binary is probably the first time it was built after it was last modified (Windows uses a lot of caching between builds to try and keep the total build time under 24 hours). So it would seem like the fix has made its way out now.
It isn't out yet. Only in development builds. Using
ucrtbase.dll19041.546 from October 13, 2020 (fully patched 20H2)(base) ➜ linearmodels git:(add-hdfe) ✗ conda activate empty (empty) ➜ linearmodels git:(add-hdfe) ✗ ipython Python 3.8.5 (default, Sep 3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)] Type 'copyright', 'credits' or 'license' for more information IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help. In [1]: import numpy as np ** On entry to DGEBAL parameter number 3 had an illegal value ** On entry to DGEHRD parameter number 2 had an illegal value ** On entry to DORGHR DORGQR parameter number 2 had an illegal value ** On entry to DHSEQR parameter number 4 had an illegal value --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-1-0aa0b027fcb6> in <module> ----> 1 import numpy as np c:\anaconda\envs\empty\lib\site-packages\numpy\__init__.py in <module> 303 304 if sys.platform == "win32" and sys.maxsize > 2**32: --> 305 _win_os_check() 306 307 del _win_os_check c:\anaconda\envs\empty\lib\site-packages\numpy\__init__.py in _win_os_check() 300 "See this issue for more information: " 301 "https://tinyurl.com/y3dm3h86") --> 302 raise RuntimeError(msg.format(__file__)) from None 303 304 if sys.platform == "win32" and sys.maxsize > 2**32: RuntimeError: The current Numpy installation ('c:\\anaconda\\envs\\empty\\lib\\site-packages\\numpy\\__init__.py') fails to pass a sanity check due to a bug in the windows runtime. See this issue for more information: https://tinyurl.com/y3dm3h86conda forge uses a newer OpenBlas, which is why @h-vetinari 's passes. The standard pip install continues to fail because the fmod patch hasn't been distributed yet.
According to the above mention, the NumPy 1.19.5 release uses a workaround for the Windows 2004 bug, instead of waiting for Microsoft to fix it.
NumPy 1.19.5 is a short bugfix release. Apart from fixing several bugs, the main improvement is the update to OpenBLAS 0.3.13 that works around the windows 2004 bug while not breaking execution on other platforms. This release supports Python 3.6-3.9 and is planned to be the last release in the 1.19.x cycle.
Reacted by Joseph S. and Hameer AbbasiThe final fix is nearing release and is now in beta and release channels. Hopefully out next month.
Windows 10 build 19042.782
- We fixed an issue that causes the 64-bit fmod() and remainder() functions to damage the Floating Point Unit (FPU) stack.
Reacted by Joseph S., Hameer Abbasi, Tobias Brückner, Tyson Linger and Sungwook Yeom- added a commit that references this issue
on Sep 20, 2021
Tests are failing:
FAILED ....\lib\tests\test_regression.py::TestRegression::test_polyfit_build - numpy.linalg.LinAlgError: SVD did not...
FAILED ....\linalg\tests\test_regression.py::TestRegression::test_eig_build - numpy.linalg.LinAlgError: Eigenvalues ...
FAILED ....\ma\tests\test_extras.py::TestPolynomial::test_polyfit - numpy.linalg.LinAlgError: SVD did not converge i...
with exceptions:
and
Steps taken:
pip install pytestpip install numpypip install hypothesisSame happens issue happens when running on tests in the repository.
Version 1.19.0 of numpy
Am I missing any dependencies? Or is it just Windows going bonkers?