GH-47367: [Packaging][Python] Patch vcpkg to show logs and install newer Windows SDK for vs_buildtools by raulcd · Pull Request #47484 · apache/arrow

raulcd · 2025-09-03T14:13:42Z

Rationale for this change

Our Windows wheels have been failing for a long time with some errors finding header files:

 Python313\Lib\subprocess.py:419: CalledProcessError
---------------------------- Captured stdout call -----------------------------
Compiling extensions.pyx because it changed.
[1/1] Cythonizing extensions.pyx
Extension module: <setuptools.extension.Extension('extensions') at 0x24849772490> ['C:\\Python313\\Lib\\site-packages\\numpy\\_core\\include', 'C:\\Python313\\Lib\\site-packages\\pyarrow\\include'] ['arrow_python', 'arrow'] ['C:\\Python313\\Lib\\site-packages\\pyarrow', 'C:\\Python313\\Lib\\site-packages\\pyarrow.libs']
running build_ext
building 'extensions' extension
creating build\temp.win-amd64-cpython-313\Release
"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Python313\Lib\site-packages\numpy\_core\include -IC:\Python313\Lib\site-packages\pyarrow\include -IC:\Python313\include -IC:\Python313\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" /EHsc /Tpextensions.cpp /Fobuild\temp.win-amd64-cpython-313\Release\extensions.obj -D_ENABLE_EXTENDED_ALIGNED_STORAGE /std:c++17
extensions.cpp
C:\Python313\include\pyconfig.h(59): fatal error C1083: Cannot open include file: 'io.h': No such file or directory

We also couldn't re-build our images because vcpkg was failing to build dependencies.

What changes are included in this PR?

When installing through vs_buildtools the different components we were using a deprecated Windows SDK. Installing VCTools with all recommended packages fixes both the vcpkg build and the test failures on the wheel but due to image size we have decided to only upgrade the Windows SDK at the moment.

Link to MSVC VCTools package:
https://learn.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools?view=vs-2022#desktop-development-with-c

In order to understand the vcpkg failure I patched vcpkg so it prints the build log error. Patching vcpkg is also part of the PR so it can be used in the future.

Are these changes tested?

Yes

Are there any user-facing changes?

No

GitHub Issue: [Python][CI] vcpkg dependencies fail rebuilding #47367

raulcd · 2025-09-03T14:14:03Z

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

github-actions · 2025-09-03T14:16:14Z

Revision: b921d55

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c0d496e90

Task	Status
wheel-windows-cp313-cp313-amd64

raulcd · 2025-09-03T15:02:18Z

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

github-actions · 2025-09-03T15:04:51Z

Revision: 534f53c

Submitted crossbow builds: ursacomputing/crossbow @ actions-674df955f6

Task	Status
wheel-windows-cp313-cp313-amd64

raulcd · 2025-09-03T15:29:41Z

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

github-actions · 2025-09-03T15:31:59Z

Revision: 8722d8b

Submitted crossbow builds: ursacomputing/crossbow @ actions-3af7390a33

Task	Status
wheel-windows-cp313-cp313-amd64

raulcd · 2025-09-03T15:33:22Z

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

github-actions · 2025-09-03T15:35:33Z

Revision: c195369

Submitted crossbow builds: ursacomputing/crossbow @ actions-d24a423afe

Task	Status
wheel-windows-cp313-cp313-amd64

raulcd · 2025-09-03T15:48:20Z

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

github-actions · 2025-09-03T15:50:34Z

Revision: 652eee8

Submitted crossbow builds: ursacomputing/crossbow @ actions-2dd7b487a4

Task	Status
wheel-windows-cp313-cp313-amd64

raulcd · 2025-09-04T06:44:53Z

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

github-actions · 2025-09-04T06:47:00Z

Revision: 4fe33ca

Submitted crossbow builds: ursacomputing/crossbow @ actions-7c37c8ca43

Task	Status
wheel-windows-cp313-cp313-amd64

raulcd · 2025-09-04T07:32:11Z

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

github-actions · 2025-09-04T07:34:21Z

Revision: 03eaa11

Submitted crossbow builds: ursacomputing/crossbow @ actions-ab25aea01a

Task	Status
wheel-windows-cp313-cp313-amd64

raulcd · 2025-09-04T08:55:52Z

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

github-actions · 2025-09-04T08:58:07Z

Revision: 29ab183

Submitted crossbow builds: ursacomputing/crossbow @ actions-435e7b2e7f

Task	Status
wheel-windows-cp313-cp313-amd64

…ldTools

raulcd · 2025-09-04T09:24:54Z

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

github-actions · 2025-09-04T09:27:06Z

Revision: 9dff6f2

Submitted crossbow builds: ursacomputing/crossbow @ actions-9b8f4ceae9

Task	Status
wheel-windows-cp313-cp313-amd64

…is required

kou

+1

kou · 2025-09-04T21:57:04Z

-    --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 `
-    --add Microsoft.VisualStudio.Component.Windows10SDK.20348 `
-    --add Microsoft.VisualStudio.Component.VC.CMake.Project `
+    --includeRecommended ^ `


I think that https://hub.docker.com/r/apache/arrow-dev/tags?name=windows&ordering=-name have the uploaded images but no images exist...

docker compose images may show the size of built images.

BTW, I think that we should use --includeRecommended even if it increases image size. We don't want to maintain Visual Studio components...

…to always apply the vcpkg patch

raulcd · 2025-09-08T07:39:54Z

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

github-actions · 2025-09-08T07:42:09Z

Revision: 8f4de06

Submitted crossbow builds: ursacomputing/crossbow @ actions-cd5b0cfb47

Task	Status
wheel-windows-cp313-cp313-amd64

raulcd · 2025-09-08T08:43:30Z

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

github-actions · 2025-09-08T08:46:46Z

Revision: 4225665

Submitted crossbow builds: ursacomputing/crossbow @ actions-ad4245e729

Task	Status
wheel-windows-cp313-cp313-amd64

pitrou

+1, thanks a lot for doing this!!

conbench-apache-arrow · 2025-09-08T21:18:48Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 64cfa7c.

There weren't enough matching historic benchmark results to make a call on whether there were regressions.

The full Conbench report has more details.

…all newer Windows SDK for vs_buildtools (apache#47484) ### Rationale for this change Our Windows wheels have been failing for a long time with some errors finding header files: ``` Python313\Lib\subprocess.py:419: CalledProcessError ---------------------------- Captured stdout call ----------------------------- Compiling extensions.pyx because it changed. [1/1] Cythonizing extensions.pyx Extension module: <setuptools.extension.Extension('extensions') at 0x24849772490> ['C:\\Python313\\Lib\\site-packages\\numpy\\_core\\include', 'C:\\Python313\\Lib\\site-packages\\pyarrow\\include'] ['arrow_python', 'arrow'] ['C:\\Python313\\Lib\\site-packages\\pyarrow', 'C:\\Python313\\Lib\\site-packages\\pyarrow.libs'] running build_ext building 'extensions' extension creating build\temp.win-amd64-cpython-313\Release "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Python313\Lib\site-packages\numpy\_core\include -IC:\Python313\Lib\site-packages\pyarrow\include -IC:\Python313\include -IC:\Python313\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" /EHsc /Tpextensions.cpp /Fobuild\temp.win-amd64-cpython-313\Release\extensions.obj -D_ENABLE_EXTENDED_ALIGNED_STORAGE /std:c++17 extensions.cpp C:\Python313\include\pyconfig.h(59): fatal error C1083: Cannot open include file: 'io.h': No such file or directory ``` We also couldn't re-build our images because vcpkg was failing to build dependencies. ### What changes are included in this PR? When installing through vs_buildtools the different components we were using a deprecated Windows SDK. Installing VCTools with all recommended packages fixes both the vcpkg build and the test failures on the wheel but due to image size we have decided to only upgrade the Windows SDK at the moment. Link to MSVC VCTools package: https://learn.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools?view=vs-2022#desktop-development-with-c In order to understand the vcpkg failure I patched vcpkg so it prints the build log error. Patching vcpkg is also part of the PR so it can be used in the future. ### Are these changes tested? Yes ### Are there any user-facing changes? No * GitHub Issue: apache#47367 Lead-authored-by: Raúl Cumplido <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

WIP: Raul testing vcpkg patch for Windows wheels

b921d55

github-actions Bot added the awaiting committer review Awaiting committer review label Sep 3, 2025

Trigger a proper rebuild

534f53c

Try pinning specific base image

8722d8b

Try a different approach too

c195369

Add extra components

652eee8

raulcd mentioned this pull request Sep 3, 2025

[Python][CI] vcpkg dependencies fail rebuilding #47367

Closed

Try new components on buildtools

4fe33ca

Include also recommended on buildtools installation

03eaa11

Install recommended, remove already installed

29ab183

Try removing Microsoft.VisualStudio.Workload.VisualStudioExtensionBui…

9dff6f2

…ldTools

Validate whether Microsoft.VisualStudio.Component.Windows10SDK.20348 …

dc08b69

…is required

kou approved these changes Sep 4, 2025

View reviewed changes

github-actions Bot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Sep 4, 2025

raulcd commented Sep 5, 2025

View reviewed changes

Comment thread ci/scripts/install_vcpkg.sh Outdated

Remove unnecessary conditional as file will always exist and we want …

4bef457

…to always apply the vcpkg patch

github-actions Bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting merge Awaiting merge awaiting changes Awaiting changes awaiting change review Awaiting change review labels Sep 5, 2025

Revert installing recommended and only update Windows SDK patch

8f4de06

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 8, 2025

raulcd changed the title ~~GH-47367: [Packaging][Python] Patch vcpkg to show logs and install VCTools with all recommended packages with vs_buildtools~~ GH-47367: [Packaging][Python] Patch vcpkg to show logs and install newer Windows SDK for vs_buildtools Sep 8, 2025

Fix patch so git does not complain

4225665

raulcd requested a review from pitrou September 8, 2025 11:11

github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 8, 2025

pitrou approved these changes Sep 8, 2025

View reviewed changes

pitrou merged commit 64cfa7c into apache:main Sep 8, 2025
35 of 36 checks passed

pitrou removed the awaiting changes Awaiting changes label Sep 8, 2025

pitrou mentioned this pull request Sep 8, 2025

Add a vcpkg install option to print out error logs on failure microsoft/vcpkg#46976

Open

Conversation

raulcd commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

raulcd commented Sep 3, 2025

Uh oh!

github-actions Bot commented Sep 3, 2025

Uh oh!

raulcd commented Sep 3, 2025

Uh oh!

github-actions Bot commented Sep 3, 2025

Uh oh!

raulcd commented Sep 3, 2025

Uh oh!

github-actions Bot commented Sep 3, 2025

Uh oh!

raulcd commented Sep 3, 2025

Uh oh!

github-actions Bot commented Sep 3, 2025

Uh oh!

raulcd commented Sep 3, 2025

Uh oh!

github-actions Bot commented Sep 3, 2025

Uh oh!

raulcd commented Sep 4, 2025

Uh oh!

github-actions Bot commented Sep 4, 2025

Uh oh!

raulcd commented Sep 4, 2025

Uh oh!

github-actions Bot commented Sep 4, 2025

Uh oh!

raulcd commented Sep 4, 2025

Uh oh!

github-actions Bot commented Sep 4, 2025

Uh oh!

raulcd commented Sep 4, 2025

Uh oh!

github-actions Bot commented Sep 4, 2025

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

kou Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

raulcd commented Sep 8, 2025

Uh oh!

github-actions Bot commented Sep 8, 2025

Uh oh!

raulcd commented Sep 8, 2025

Uh oh!

github-actions Bot commented Sep 8, 2025

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

conbench-apache-arrow Bot commented Sep 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

raulcd commented Sep 3, 2025 •

edited

Loading