Repository navigation
[Python] Nightly wheels missing for Windows #47308
Description
Activity
- changed the title
[-]Nightly wheels missing for Windows[/-][+][Python] Nightly wheels missing for Windows[/+]on Aug 11, 2025 Thanks for raising the issue. There's a maximum number of artifacts retained that removes old wheels if they fail for ~4-5 days. The Windows wheels nightly jobs have been failing for 6 days. It requires some investigation, I won't be available to investigate for the rest of the week but the failures are due to some failures on 3 cython tests. If someone can investigate:
Failures example:
https://github.com/ursacomputing/crossbow/actions/runs/16745071346/job/47401564422I plan to do some investigation tomorrow (alongside the free-threaded build failures) and will share what I find.
A new release of Visual Studio 2022 might be causing issues. See:
-
A note regarding the Windows SDK 10.0.20348 we use, which has been removed from the VS 2022 installer in the above version:
The following Windows SDK versions have been removed from the Visual Studio 2022 installer: 10.0.18362.0, 10.0.20348.0 and 10.0.22000.0. If you previously installed one of these versions of the SDK using Visual Studio it will be uninstalled when you update. If your project targets any of these SDKs you may encounter a build error such as: The Windows SDK version 10.0.22000.0 was not found. Install the required version of Windows SDK or change the SDK version in the project property pages or by right-clicking the solution and selecting "Retarget solution". To resolve this, we recommend retargeting your project to 10.0.26100.0, or an earlier supported version if necessary. For a complete list of supported SDK versions please visit: https://developer.microsoft.com/windows/downloads/sdk-archive/. If you need to install an unsupported version of the SDK, you can find it here: https://developer.microsoft.com/windows/downloads/sdk-archive/index-legacy.
I plan to try updating the Python Windows wheel Dockerfiles in this PR to see if that helps.
I have built PyArrow on a Windows machine and cannot reproduce the failure. I should have used the same MSVC, flags and Python version. The main difference I see is in the include paths.
Locally, the
ucrt,um,shared, and other Windows Kits directories are present. But in the CI, these are missing — only...\includefolders appear.Local build output
(pyarrow-dev) C:\Users\Public\Repositories\arrow\python>python -m pytest pyarrow/tests/test_cython.py -k test_cython_api -s Package "tzdata" not found. Not setting TZDIR environment variable. ============================================ test session starts ============================================ platform win32 -- Python 3.12.11, pytest-8.4.1, pluggy-1.6.0 rootdir: C:\Users\Public\Repositories\arrow\python configfile: setup.cfg plugins: hypothesis-6.138.0 collected 2 items / 1 deselected / 1 selected pyarrow\tests\test_cython.py Compiling pyarrow_cython_example.pyx because it changed. [1/1] Cythonizing pyarrow_cython_example.pyx Extension module: <setuptools.extension.Extension('pyarrow_cython_example') at 0x2a89e690d40> ['C:\\Users\\Inti\\.conda\\envs\\pyarrow-dev\\Lib\\site-packages\\numpy\\_core\\include', 'C:\\Users\\Public\\Repositories\\arrow\\python\\pyarrow\\include'] ['arrow_python', 'arrow'] ['C:\\Users\\Public\\Repositories\\arrow\\python\\pyarrow', 'C:/Users/Inti/.conda/envs/pyarrow-dev/Library/lib', 'C:\\Users\\Inti\\.conda\\envs\\pyarrow-dev\\Library\\lib'] running build_ext building 'pyarrow_cython_example' extension creating build\temp.win-amd64-cpython-312\Release "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Users\Inti\.conda\envs\pyarrow-dev\Lib\site-packages\numpy\_core\include -IC:\Users\Public\Repositories\arrow\python\pyarrow\include -IC:\Users\Inti\.conda\envs\pyarrow-dev\include -IC:\Users\Inti\.conda\envs\pyarrow-dev\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\cppwinrt" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.26100.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.26100.0\\cppwinrt" /EHsc /Tppyarrow_cython_example.cpp /Fobuild\temp.win-amd64-cpython-312\Release\pyarrow_cython_example.obj -D_ENABLE_EXTENDED_ALIGNED_STORAGE /std:c++17 pyarrow_cython_example.cpp pyarrow_cython_example.cpp(13990): warning C4551: function call missing argument list pyarrow_cython_example.cpp(13997): warning C4551: function call missing argument list creating C:\Users\Inti\AppData\Local\Temp\pytest-of-Inti\pytest-1\test_cython_api0\build\lib.win-amd64-cpython-312 "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\HostX86\x64\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\Public\Repositories\arrow\python\pyarrow /LIBPATH:C:/Users/Inti/.conda/envs/pyarrow-dev/Library/lib /LIBPATH:C:\Users\Inti\.conda\envs\pyarrow-dev\Library\lib /LIBPATH:C:\Users\Inti\.conda\envs\pyarrow-dev\libs /LIBPATH:C:\Users\Inti\.conda\envs\pyarrow-dev /LIBPATH:C:\Users\Inti\.conda\envs\pyarrow-dev\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.26100.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\\lib\10.0.26100.0\\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.26100.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\\lib\10.0.26100.0\\um\x64" arrow_python.lib arrow.lib /EXPORT:PyInit_pyarrow_cython_example build\temp.win-amd64-cpython-312\Release\pyarrow_cython_example.obj /OUT:build\lib.win-amd64-cpython-312\pyarrow_cython_example.cp312-win_amd64.pyd /IMPLIB:build\temp.win-amd64-cpython-312\Release\pyarrow_cython_example.cp312-win_amd64.lib Creating library build\temp.win-amd64-cpython-312\Release\pyarrow_cython_example.cp312-win_amd64.lib and object build\temp.win-amd64-cpython-312\Release\pyarrow_cython_example.cp312-win_amd64.exp Generating code Finished generating code copying build\lib.win-amd64-cpython-312\pyarrow_cython_example.cp312-win_amd64.pyd -> . ====================================== 1 passed, 1 deselected in 9.80s ======================================CI build output
"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Python312\Lib\site-packages\numpy\core\include -IC:\Python312\Lib\site-packages\pyarrow\include -IC:\Python312\include -IC:\Python312\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" /EHsc /Tppyarrow_cython_example.cpp /Fobuild\temp.win-amd64-cpython-312\Release\pyarrow_cython_example.obj -D_ENABLE_EXTENDED_ALIGNED_STORAGE /std:c++17 pyarrow_cython_example.cpp C:\Python312\include\pyconfig.h(59): fatal error C1083: Cannot open include file: 'io.h': No such file or directoryThis would suggests a problem with the Visual Studio environment on the CI— likely the Windows SDK is missing or not configured properly.
See:
- discuss.python.org: Where is io.h file used as an include in pyconfig.h (Python 3.12)
- Microsoft Q&A: io.h not found
Interestingly, C++ and PyArrow build successfully, but the Cython test fails, which is unexpected. Currently the only idea I have is to skip tests in case the header files are not found/Windows SDK is not installed.
Thought at first that delvewheel versions were the same beteween builds but now I see there has been a bump:
- failing CI build https://github.com/ursacomputing/crossbow/actions/runs/17018910791/job/48245182645#step:7:7904
delvewheel-1.11.1 pefile-2024.8.26 - successful CI build https://github.com/ursacomputing/crossbow/actions/runs/16718301087/job/47316555137#step:7:7905
installed delvewheel-1.11.0 pefile-2024.8.26
the timing matches (5th of August) but the changelog shows nothing that would be connected:
https://github.com/adang1345/delvewheel/blob/master/CHANGELOG.md#1111-5-august-2025
- Miscellaneous README improvements.
- Fix UnicodeDecodeError when patching .py files containing Unicode characters.
- Don't strip off ending newline when inserting .py patch after docstring.
- failing CI build https://github.com/ursacomputing/crossbow/actions/runs/17018910791/job/48245182645#step:7:7904
Windows wheels have been uploaded successfully:
https://anaconda.org/scientific-python-nightly-wheels/pyarrow/files
Describe the bug, including details regarding any error messages, version, and platform.
See
https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/pyarrow/
This suggests they were either removed on purpose (sorry if I missed the discussion!) or that the upload hasn't worked for about a month. Or at least I think ~1mo is how long the artifacts are retained there...
Component(s)
Python