[SPARK-59920][BUILD] Remove unnecessary git clean from pip tests script - #59182

The reasoning is sound, but the two mechanisms don't actually free the same space, so "we already free space via free-disk-space" doesn't by itself make git clean redundant:

• free-disk-space runs dev/free_disk_space / dev/free_disk_space_container, which delete runner/container pre-installed bulk (dotnet, Android SDK, large Docker images, etc.) to increase total free space on the host/container.
• The removed git clean -d -f -x deleted untracked/ignored build artifacts inside the Spark working tree.

These target different sources of disk pressure, so one isn't a drop-in replacement for the other. The stronger (and sufficient) justification is the one you already give alongside it: the runners simply have much more free space now, and (because the pip job starts from a fresh checkout and the existing CI run confirms it) the working-tree cleanup is no longer needed for the pip test to fit on disk.

Suggestion: reword to something fact-based, e.g.:

The runners have much more free space now, and the pip job already starts from a fresh checkout, so the pip test fits on disk without this working-tree cleanup (confirmed by existing CI). Note free-disk-space frees host/container space rather than the working tree, so it isn't a direct substitute; the point is that this cleanup simply isn't needed anymore.

This is purely a description nit; the code change itself is fine.

Thanks for the cleanup. Removing a CI environment-prep step from the test script itself makes sense to me. However, I'd like the following to be addressed (or clarified) before merging.

1. The macOS hosted-runner job loses its only disk cleanup

The PR description justifies the removal with the free-disk-space parameter of setup-build-env. python_hosted_runner_test.yml (default macos-15, used by build_python_3.12_arm.yml and build_python_3.12_macos26.yml) calls setup-build-env without free-disk-space, and the host/container scripts are Linux-only (dpkg, sudo rm -rf /usr/share/dotnet, ...). That workflow sets SKIP_PACKAGING=false for pyspark-errors, so dev/run-tests still invokes run-pip-tests there. The removed git clean was gated on GITHUB_ACTIONS, so it did run on those runners, and after this PR nothing replaces it. The runner-images link in the description is about Linux runners.

Could you confirm the macOS runners have enough headroom for the precompiled artifact (every module target/) plus two venvs and two PySpark installs? If not, please add an explicit cleanup step to that workflow rather than dropping it.

2. The script no longer guarantees a clean tree

git clean -d -f -x also removed ignored/untracked leftovers such as python/dist, python/build and generated *.json files. The script still hard-fails with Unexpected number of targets found in dist directory - please cleanup existing sdists first if python/dist has more than one sdist, and setup.py sdist uses include_package_data=True plus pyspark/**/*.json globs. On a fresh GitHub checkout this is fine. On a persistent or self-hosted workspace, or a re-run after a packaging run with a different version, it can now fail or ship stray files. If that is intentionally out of scope, a one-line note in the description would be enough.

3. Validation

The description says "Existing CI", but run-pip-tests only runs when SKIP_PACKAGING=false, i.e. for the pyspark-periodic module in build_and_test.yml (excluded unless pyspark-periodic is required) and pyspark-errors in python_hosted_runner_test.yml. A PR that only touches dev/run-pip-tests may not select either, so a green CI would not show that the pip tests still pass without the cleanup. Could you link a run where the pip packaging step actually executed with this change (ideally on both Linux and macOS)?