Repository navigation
TestLiveSigstoreVerifier/with_2/3_verified_attestations is flaky #10390
Description
Activity
@cli/package-security for help investigating this one 🙏
Reacted by Meredith LancasterGetting the same error for Windows:
--- FAIL: TestVerifyAttestations (1.43s) --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (0.41s) attestation_integration_test.go:53: Error Trace: D:/a/cli/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:53 Error: "[0xc00090fdf0]" should have 2 item(s), but has 1 Test: TestVerifyAttestations/all_attestations_pass_verificationWorkflow: https://github.com/cli/cli/actions/runs/13251671571/job/36990690943?pr=10334#step:5:61
Here's another one:
--- FAIL: TestVerifyAttestations (1.77s) --- FAIL: TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification (0.39s) attestation_integration_test.go:65: Error Trace: D:/a/cli/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:65 Error: "[0xc000391ef0]" should have 2 item(s), but has 1 Test: TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verificationWorkflow: https://github.com/cli/cli/actions/runs/13258224226/job/37008805848?pr=10413#step:5:61
2025-02-12 (Ubuntu)
--- FAIL: TestVerifyIntegration (18.63s) --- FAIL: TestVerifyIntegration/with_invalid_owner (15.17s) verify_integration_test.go:76: Error Trace: /home/runner/work/cli/cli/pkg/cmd/attestation/verify/verify_integration_test.go:76 Error: Error "failed to find recognized issuer from bundle content: failed to create TUF client: tuf refresh failed: context deadline exceeded (Client.Timeout or context cancellation while reading body)" does not contain "expected SourceRepositoryOwnerURI to be https://github.com/fakeowner, got https://github.com/sigstore" Test: TestVerifyIntegration/with_invalid_ownerWorkflow: https://github.com/cli/cli/actions/runs/13288202067/job/37102016536#step:5:64
2025-02-13 (macOS)
WARNING: DATA RACE Read at 0x00c000440ae0 by goroutine 72: github.com/cli/cli/v2/pkg/cmd/attestation/api.(*failAfterNCallsHttpClient).Get() /Users/runner/work/cli/cli/pkg/cmd/attestation/api/mock_httpClient_test.go:69 +0x68 github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).getBundle.func1() /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:194 +0x78 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1() /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:18 +0x30 github.com/cenkalti/backoff/v4.doRetryNotify[go.shape.struct {}]() /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:88 +0x15c github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:61 +0x80 github.com/cenkalti/backoff/v4.RetryNotify() /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:49 +0x1b0 github.com/cenkalti/backoff/v4.Retry() /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:38 +0x190 github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).getBundle() /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:193 +0x138 github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).fetchBundleFromAttestations.func1() /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:169 +0x1f0 golang.org/x/sync/errgroup.(*Group).Go.func1() /Users/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x7c Previous write at 0x00c000440ae0 by goroutine 73: github.com/cli/cli/v2/pkg/cmd/attestation/api.(*failAfterNCallsHttpClient).Get() /Users/runner/work/cli/cli/pkg/cmd/attestation/api/mock_httpClient_test.go:69 +0x7c github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).getBundle.func1() /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:194 +0x78 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1() /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:18 +0x30 github.com/cenkalti/backoff/v4.doRetryNotify[go.shape.struct {}]() /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:88 +0x15c github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:61 +0x80 github.com/cenkalti/backoff/v4.RetryNotify() /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:49 +0x1b0 github.com/cenkalti/backoff/v4.Retry() /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:38 +0x190 github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).getBundle() /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:193 +0x138 github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).fetchBundleFromAttestations.func1() /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:169 +0x1f0 golang.org/x/sync/errgroup.(*Group).Go.func1() /Users/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x7c Goroutine 72 (running) created at: golang.org/x/sync/errgroup.(*Group).Go() /Users/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x10c github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).fetchBundleFromAttestations() /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:1[54](https://github.com/cli/cli/actions/runs/13301543234/job/37143755142?pr=10430#step:5:55) +0xb4 github.com/cli/cli/v2/pkg/cmd/attestation/api.TestFetchBundleFromAttestations_FailOnTheSecondAttestation() /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client_test.go:215 +0x2e0 testing.tRunner() /Users/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1690 +0x184 testing.(*T).Run.gowrap1() /Users/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1743 +0x40 Goroutine 73 (running) created at: golang.org/x/sync/errgroup.(*Group).Go() /Users/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x10c github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).fetchBundleFromAttestations() /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:154 +0xb4 github.com/cli/cli/v2/pkg/cmd/attestation/api.TestFetchBundleFromAttestations_FailOnTheSecondAttestation() /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client_test.go:215 +0x2e0 testing.tRunner() /Users/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1690 +0x184 testing.(*T).Run.gowrap1() /Users/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1743 +0x40 ================== --- FAIL: TestFetchBundleFromAttestations_FailOnTheSecondAttestation (0.[63](https://github.com/cli/cli/actions/runs/13301543234/job/37143755142?pr=10430#step:5:64)s) testing.go:1399: race detected during execution of test FAILWorkflow: https://github.com/cli/cli/actions/runs/13301543234/job/37143755142?pr=10430#step:5:54
Thanks for the report, the team is taking a look into this
Reacted by Josh SorefThis should be resolved by #10439 but please let us know if you run into this problem again.
Reacted by Josh Soref@malancas: It's still failing:
--- FAIL: TestLiveSigstoreVerifier (2.24s) --- FAIL: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.42s) sigstore_integration_test.go:79: Error Trace: D:/a/cli/cli/pkg/cmd/attestation/verification/sigstore_integration_test.go:79 Error: "[0xc0002c0780]" should have 2 item(s), but has 1 Test: TestLiveSigstoreVerifier/with_2/3_verified_attestations FAILWorkflow: https://github.com/iamazeem/cli/actions/runs/13334014863/job/37244936343#step:5:60
2025-02-16 (Windows)
--- FAIL: TestLiveSigstoreVerifier (1.98s) --- FAIL: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.34s) sigstore_integration_test.go:79: Error Trace: D:/a/cli/cli/pkg/cmd/attestation/verification/sigstore_integration_test.go:79 Error: "[0xc0004b3430]" should have 2 item(s), but has 1 Test: TestLiveSigstoreVerifier/with_2/3_verified_attestationsWorkflow: https://github.com/cli/cli/actions/runs/13354330624/job/37294953903?pr=10456#step:5:59
2025-02-18 (Windows)
--- FAIL: TestLiveSigstoreVerifier (3.02s) --- FAIL: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.48s) sigstore_integration_test.go:79: Error Trace: D:/a/cli/cli/pkg/cmd/attestation/verification/sigstore_integration_test.go:79 Error: "[0xc00018b8c0]" should have 2 item(s), but has 1 Test: TestLiveSigstoreVerifier/with_2/3_verified_attestations FAIL FAIL github.com/cli/cli/v2/pkg/cmd/attestation/verification 3.302s ? github.com/cli/cli/v2/pkg/cmd/auth/shared/contract [no test files] --- FAIL: TestVerifyAttestations (1.80s) --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (0.61s) attestation_integration_test.go:53: Error Trace: D:/a/cli/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:53 Error: "[0xc0009265[60](https://github.com/cli/cli/actions/runs/13383511075/job/37376071790?pr=10456#step:5:61)]" should have 2 item(s), but has 1 Test: TestVerifyAttestations/all_attestations_pass_verification FAILWorkflow: https://github.com/cli/cli/actions/runs/13383511075/job/37376071790?pr=10456#step:5:59
Reacted by Meredith LancasterReacted by Josh Soref- Reacted by Azeem
UPDATE
Running tests locally on Windows 10 also reproduces this.
However, it's a bit random but overall percentage is pretty high i.e. ~80%.Command:
go test -tags=integration .\pkg\cmd\attestation\... -count 2 -failfastErrors:
--- FAIL: TestVerifyAttestations (1.13s) --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (1.12s) attestation_integration_test.go:53: Error Trace: C:/cli-trunk/pkg/cmd/attestation/verify/attestation_integration_test.go:53 Error: "[0xc000719230]" should have 2 item(s), but has 1 Test: TestVerifyAttestations/all_attestations_pass_verification--- FAIL: TestLiveSigstoreVerifier (3.09s) sigstore_integration_test.go:62: Error Trace: C:/cli-trunk/pkg/cmd/attestation/verification/sigstore_integration_test.go:62 Error: Not equal: expected: 2 actual : 1 Test: TestLiveSigstoreVerifier Messages: test case: with valid artifact and JSON lines file containing multiple Sigstore bundles--- FAIL: TestVerifyAttestations (3.72s) --- FAIL: TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification (1.42s) attestation_integration_test.go:65: Error Trace: C:/cli-trunk/pkg/cmd/attestation/verify/attestation_integration_test.go:65 Error: "[0xc000781d30]" should have 2 item(s), but has 1 Test: TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification--- FAIL: TestVerifyAttestations (1.85s) --- FAIL: TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification (0.18s) attestation_integration_test.go:63: Error Trace: C:/cli-trunk/pkg/cmd/attestation/verify/attestation_integration_test.go:63 Error: Received unexpected error: failed to get bundle issuer: unsupported bundle version: application/vnd.dev.sigstore.bundle+json;version=0.1 Test: TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification--- FAIL: TestLiveSigstoreVerifier (3.61s) --- FAIL: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.82s) sigstore_integration_test.go:79: Error Trace: C:/cli-trunk/pkg/cmd/attestation/verification/sigstore_integration_test.go:79 Error: "[0xc000293310]" should have 2 item(s), but has 1 Test: TestLiveSigstoreVerifier/with_2/3_verified_attestations
Following two JSON files are generated by running these tests:
pkg/cmd/attestation/verification/tuf-repo.github.com.jsonpkg/cmd/attestation/verification/tuf-repo-cdn.sigstore.dev.json
and are not removed automatically.
Looks like they may be removed when the tests finish and/or added to.gitignore.@williammartin: I'm running these tests in Powershell on Windows 10 Home.
Besides, these tests are failing in CI pipeline also as mentioned above.Besides, these tests are failing in CI pipeline also as mentioned #10390 (comment).
Sure, I'm just trying to reproduce it myself to think about fixing it. Thanks.
Reacted by Azeem- addedtech-debtA chore that addresses technical debtA chore that addresses technical debtand removedbugSomething isn't workingSomething isn't workingneeds-triageneeds to be reviewedneeds to be reviewed
on Feb 19, 2025 FWIW this branch contains some modifications that I believe only run 2 tests, and it fails very frequently (after I bumped the specs on my Windows VM):
PS C:\Users\williammartin\workspace\cli> go test -v -tags=integration ./pkg/cmd/attestation/verification ./pkg/cmd/attestation/verify -count 2 -failfast === RUN TestLiveSigstoreVerifier === RUN TestLiveSigstoreVerifier/with_2/3_verified_attestations --- PASS: TestLiveSigstoreVerifier (0.78s) --- PASS: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.78s) === RUN TestLiveSigstoreVerifier === RUN TestLiveSigstoreVerifier/with_2/3_verified_attestations --- PASS: TestLiveSigstoreVerifier (0.59s) --- PASS: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.59s) PASS ok github.com/cli/cli/v2/pkg/cmd/attestation/verification 1.499s === RUN TestVerifyAttestations === RUN TestVerifyAttestations/all_attestations_pass_verification attestation_integration_test.go:53: Error Trace: C:/Users/williammartin/workspace/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:53 Error: "[0x400045ae30]" should have 2 item(s), but has 1 Test: TestVerifyAttestations/all_attestations_pass_verification --- FAIL: TestVerifyAttestations (0.43s) --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (0.43s) FAIL FAIL github.com/cli/cli/v2/pkg/cmd/attestation/verify 0.548s FAILReacted by Meredith LancasterI believe this is occurring because instantiation of the TUF Client results in a json file being written to disk via a
renameoperation, and this is not concurrent safe on Windows: https://github.com/theupdateframework/go-tuf/blob/f59c91f01f8a64466d0ecbe0cd303469615de1e9/metadata/updater/updater.go#L628Here's the error I eventually managed to get out of the tests:
sigstore.go:227: FAIL: verifying: failed to find recognized issuer from bundle content: failed to create TUF client: tuf refresh failed: rename C:\Users\williammartin\AppData\Local\GitHub CLI\.sigstore\root\tuf-repo-cdn.sigstore.dev\tuf_tmp3512682291 C:\Users\williammartin\AppData\Local\GitHub CLI\.sigstore\root\tuf-repo-cdn.sigstore.dev\root.json: Access is denied.From the windows godoc:
// Rename renames (moves) oldpath to newpath. // If newpath already exists and is not a directory, Rename replaces it. // OS-specific restrictions may apply when oldpath and newpath are in different directories. // Even within the same directory, on non-Unix platforms Rename is not an atomic operation. <-------- // If there is an error, it will be of type *LinkError.Some further validation, when I run the tests with
CODESPACES=true, I cannot reproduce this flake. This makes sense because this env var results in us setting a TUF client config option that results in the file writing being skipped.I think #10478 should resolve this but will reopen if we see these flakes reappear in CI.
Describe the bug
TestLiveSigstoreVerifier/with_2/3_verified_attestations occasionally fails
Affected version
PRs to trunk
Steps to reproduce the behavior
Expected vs actual behavior
Expected: pass
Actual:
https://github.com/cli/cli/actions/workflows/go.yml?query=is%3Afailure
windows-latest
https://github.com/cli/cli/actions/runs/13184622557/job/36803634377#step:5:59
https://github.com/cli/cli/actions/runs/13202858867/job/36858853456#step:5:61
macos-latest
https://github.com/cli/cli/actions/runs/12916504935/job/36020835039#step:5:60
ubuntu-latest
https://github.com/cli/cli/actions/runs/12680772503/job/35343196076#step:5:64
https://github.com/cli/cli/actions/runs/12659904205/job/35279973709#step:5:64
Logs
Paste the activity from your command line. Redact if needed.