Repository navigation
[SPARK-44557][INFRA] Clean up untracked/ignored files before running pip packaging test in GitHub Actions - #42159
Closed
HyukjinKwon wants to merge 12 commits into
Closed
HyukjinKwon wants to merge 12 commits into
HyukjinKwon wants to merge 12 commits into
Conversation
HyukjinKwon
marked this pull request as draft
July 26, 2023 00:36
HyukjinKwon
changed the title
HyukjinKwon
marked this pull request as ready for review
July 26, 2023 11:39
zhengruifeng
approved these changes
Jul 26, 2023
HyukjinKwon
marked this pull request as draft
July 26, 2023 22:38
HyukjinKwon
marked this pull request as ready for review
July 27, 2023 01:57
HyukjinKwon
force-pushed
the
debug-ci-failure
branch
from
July 27, 2023 03:42
3b41bf2 to
42fede5
Compare
Member
Author
|
Merged to master, branch-3.5, and branch-3.4. |
HyukjinKwon
added a commit
that referenced
this pull request
Jul 27, 2023
…pip packaging test in GitHub Actions ### What changes were proposed in this pull request? This PR proposes to remove untracked/ignored files before running pip packaging test in GitHub Actions. ### Why are the changes needed? In order to fix the flakiness in the test such as: ``` ... creating dist Creating tar archive error: [Errno 28] No space left on device Cleaning up temporary directory - /tmp/tmp.CvSzgB7Kyy ``` See also https://github.com/apache/spark/actions/runs/5665869112/job/15351515539. ### Does this PR introduce _any_ user-facing change? No, test-only. ### How was this patch tested? GitHub Actions build in this PR. Closes #42159 from HyukjinKwon/debug-ci-failure. Lead-authored-by: Hyukjin Kwon <[email protected]> Co-authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]> (cherry picked from commit 77aab6e) Signed-off-by: Hyukjin Kwon <[email protected]>
HyukjinKwon
added a commit
that referenced
this pull request
Jul 27, 2023
…pip packaging test in GitHub Actions ### What changes were proposed in this pull request? This PR proposes to remove untracked/ignored files before running pip packaging test in GitHub Actions. ### Why are the changes needed? In order to fix the flakiness in the test such as: ``` ... creating dist Creating tar archive error: [Errno 28] No space left on device Cleaning up temporary directory - /tmp/tmp.CvSzgB7Kyy ``` See also https://github.com/apache/spark/actions/runs/5665869112/job/15351515539. ### Does this PR introduce _any_ user-facing change? No, test-only. ### How was this patch tested? GitHub Actions build in this PR. Closes #42159 from HyukjinKwon/debug-ci-failure. Lead-authored-by: Hyukjin Kwon <[email protected]> Co-authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]> (cherry picked from commit 77aab6e) Signed-off-by: Hyukjin Kwon <[email protected]>
viirya
pushed a commit
to viirya/spark-1
that referenced
this pull request
Oct 19, 2023
…pip packaging test in GitHub Actions ### What changes were proposed in this pull request? This PR proposes to remove untracked/ignored files before running pip packaging test in GitHub Actions. ### Why are the changes needed? In order to fix the flakiness in the test such as: ``` ... creating dist Creating tar archive error: [Errno 28] No space left on device Cleaning up temporary directory - /tmp/tmp.CvSzgB7Kyy ``` See also https://github.com/apache/spark/actions/runs/5665869112/job/15351515539. ### Does this PR introduce _any_ user-facing change? No, test-only. ### How was this patch tested? GitHub Actions build in this PR. Closes apache#42159 from HyukjinKwon/debug-ci-failure. Lead-authored-by: Hyukjin Kwon <[email protected]> Co-authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]> (cherry picked from commit 77aab6e) Signed-off-by: Hyukjin Kwon <[email protected]>
nchammas
added a commit
that referenced
this pull request
Oct 5, 2026
…ript ### What changes were proposed in this pull request? Remove the `git clean` step from `run-pip-tests`. ### Why are the changes needed? This cleanup was added in #42159 to help resolve intermittent space pressure. It had no other purpose. It's not necessary anymore. We already free some space on the runner via the `free-disk-space` parameter to the `setup-build-env` composite action. More importantly, the runners themselves also have [much more free space][1] than in prior years when this fix was added. [1]: actions/runner-images#14492 (comment) Looking forward, this cleanup will also make it easier to change when or how the pip tests are triggered (as part of SPARK-59180), since we no longer have to worry about `git clean` removing build artifacts that other tests may need to run. Finally, an environment preparation step like this, should we ever need to reintroduce it for some reason, doesn't belong inside the test script. It should happen upstream of it, so as not to mix in logically unrelated steps with the actual pip tests themselves. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing CI for this PR [successfully ran the pip tests][ci]. [ci]: https://github.com/nchammas/spark/actions/runs/36834488478/job/110298902984#step:9:130 I also pushed a temporary commit at 0ce4a1b to get the macOS 26 build to run and confirmed that the runner has [plenty of space available][mac]. [mac]: https://github.com/nchammas/spark/actions/runs/36988944346/job/110793319313#step:2:29 ### Was this patch authored or co-authored using generative AI tooling? No. Closes #59182 from nchammas/py-packaging-test-cleanup. Authored-by: Nicholas Chammas <[email protected]> Signed-off-by: Nicholas Chammas <[email protected]>
nchammas
added a commit
that referenced
this pull request
Oct 5, 2026
…ript ### What changes were proposed in this pull request? Remove the `git clean` step from `run-pip-tests`. ### Why are the changes needed? This cleanup was added in #42159 to help resolve intermittent space pressure. It had no other purpose. It's not necessary anymore. We already free some space on the runner via the `free-disk-space` parameter to the `setup-build-env` composite action. More importantly, the runners themselves also have [much more free space][1] than in prior years when this fix was added. [1]: actions/runner-images#14492 (comment) Looking forward, this cleanup will also make it easier to change when or how the pip tests are triggered (as part of SPARK-59180), since we no longer have to worry about `git clean` removing build artifacts that other tests may need to run. Finally, an environment preparation step like this, should we ever need to reintroduce it for some reason, doesn't belong inside the test script. It should happen upstream of it, so as not to mix in logically unrelated steps with the actual pip tests themselves. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing CI for this PR [successfully ran the pip tests][ci]. [ci]: https://github.com/nchammas/spark/actions/runs/36834488478/job/110298902984#step:9:130 I also pushed a temporary commit at 0ce4a1b to get the macOS 26 build to run and confirmed that the runner has [plenty of space available][mac]. [mac]: https://github.com/nchammas/spark/actions/runs/36988944346/job/110793319313#step:2:29 ### Was this patch authored or co-authored using generative AI tooling? No. Closes #59182 from nchammas/py-packaging-test-cleanup. Authored-by: Nicholas Chammas <[email protected]> Signed-off-by: Nicholas Chammas <[email protected]> (cherry picked from commit b93fb8d) Signed-off-by: Nicholas Chammas <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR proposes to remove untracked/ignored files before running pip packaging test in GitHub Actions.
Why are the changes needed?
In order to fix the flakiness in the test such as:
See also https://github.com/apache/spark/actions/runs/5665869112/job/15351515539.
Does this PR introduce any user-facing change?
No, test-only.
How was this patch tested?
GitHub Actions build in this PR.