Skip to content

fix(IDX): Don't overwrite CDN artifacts - #5438

Merged
nmattia merged 2 commits into
masterfrom
nm-immutable-upload
Jun 12, 2025
Merged

nmattia merged 2 commits into
masterfrom
nm-immutable-upload

Conversation

@nmattia

@nmattia nmattia commented Jun 5, 2025

Copy link
Copy Markdown
Contributor

We use --immutable to avoid overwriting artifacts in the CDN/S3. Unfortunately, for single-file uploads, --immutable does not work as expected: rclone/rclone#4921

This introduces a workaround so that we can still upload artifacts one by one -- instead of full directories -- as uploading artifacts individually allows us to produce meaningful logs.

@nmattia nmattia added the CI_ALL_BAZEL_TARGETS Runs all bazel targets label Jun 5, 2025
@github-actions github-actions Bot added the fix label Jun 5, 2025
@nmattia
nmattia force-pushed the nm-immutable-upload branch from 0d30c09 to 7a90867 Compare June 5, 2025 14:42
@nmattia

nmattia commented Jun 5, 2025

Copy link
Copy Markdown
Contributor Author

Seems to work!

-- uploading 'SHA256SUMS' (remote path: 'ic/ce3a4a2c359ca00198b29ac4f140a718f6d4a838/binaries/x86_64-darwin/SHA256SUMS') --
uploading to 'ic/ce3a4a2c359ca00198b29ac4f140a718f6d4a838/binaries/x86_64-darwin/SHA256SUMS' ('/private/var/tmp/bazel-output/execroot/_main/bazel-out/darwin_x86_64-opt/bin/publish/binaries/bundle-bundle-sns-testing/binaries/x86_64-darwin/SHA256SUMS' -> 'ic/ce3a4a2c359ca00198b29ac4f140a718f6d4a838/binaries/x86_64-darwin')
uploading to AWS
2025/06/05 17:41:25 ERROR : SHA256SUMS: Source and destination exist but do not match: immutable file modified
2025/06/05 17:41:25 ERROR : Can't retry any of the errors - not attempting retries
2025/06/05 17:41:25 NOTICE: Failed to copy: immutable file modified

Now need to figure out why the SHA256SUMS are changing...

@nmattia
nmattia marked this pull request as ready for review June 5, 2025 15:50
@nmattia
nmattia requested a review from a team as a code owner June 5, 2025 15:50
@github-actions github-actions Bot added the @idx label Jun 5, 2025
We use `--immutable` to avoid overwriting artifacts in the CDN/S3.
Unfortunately, for single-file uploads, `--immutable` does not work as
expected: rclone/rclone#4921

This introduces a workaround so that we can still upload artifacts one
by one -- instead of full directories -- as uploading artifacts
individually allows us to produce meaningful logs.
@nmattia
nmattia force-pushed the nm-immutable-upload branch from ad50388 to 6c74997 Compare June 10, 2025 14:18
@nmattia

nmattia commented Jun 10, 2025

Copy link
Copy Markdown
Contributor Author

This is the cause: #5487

@nmattia
nmattia enabled auto-merge June 12, 2025 23:09
@nmattia nmattia removed the CI_ALL_BAZEL_TARGETS Runs all bazel targets label Jun 12, 2025
@nmattia
nmattia disabled auto-merge June 12, 2025 23:11
@nmattia
nmattia force-pushed the nm-immutable-upload branch from dd7bae5 to 465b7dc Compare June 12, 2025 23:11
@nmattia
nmattia enabled auto-merge June 12, 2025 23:11
@nmattia
nmattia added this pull request to the merge queue Jun 12, 2025
Merged via the queue into master with commit 12afb25 Jun 12, 2025
@nmattia
nmattia deleted the nm-immutable-upload branch June 12, 2025 23:29
aterga pushed a commit that referenced this pull request Jun 17, 2025
We use `--immutable` to avoid overwriting artifacts in the CDN/S3.
Unfortunately, for single-file uploads, `--immutable` does not work as
expected: rclone/rclone#4921

This introduces a workaround so that we can still upload artifacts one
by one -- instead of full directories -- as uploading artifacts
individually allows us to produce meaningful logs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants