Skip to content

TestLiveSigstoreVerifier/with_2/3_verified_attestations is flaky #10390

Description

@jsoref

Describe the bug

TestLiveSigstoreVerifier/with_2/3_verified_attestations occasionally fails

Affected version

PRs to trunk

Steps to reproduce the behavior

  1. Type this '...'
  2. View the output '....'
  3. See error

Expected vs actual behavior

Expected: pass

Actual:

https://github.com/cli/cli/actions/workflows/go.yml?query=is%3Afailure

windows-latest

https://github.com/cli/cli/actions/runs/13184622557/job/36803634377#step:5:59


--- FAIL: TestLiveSigstoreVerifier (3.12s)
    --- FAIL: TestLiveSigstoreVerifier/with_2/3_verified_attestations (1.12s)
        sigstore_integration_test.go:79: 
            	Error Trace:	D:/a/cli/cli/pkg/cmd/attestation/verification/sigstore_integration_test.go:79
            	Error:      	"[0xc00004b0f0]" should have 2 item(s), but has 1
            	Test:       	TestLiveSigstoreVerifier/with_2/3_verified_attestations
FAIL
FAIL	github.com/cli/cli/v2/pkg/cmd/attestation/verification	3.468s

https://github.com/cli/cli/actions/runs/13202858867/job/36858853456#step:5:61

--- FAIL: TestVerifyAttestations (1.78s)
    --- FAIL: TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification (0.37s)
        attestation_integration_test.go:65: 
            	Error Trace:	D:/a/cli/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:65
            	Error:      	"[0xc000875820]" should have 2 item(s), but has 1
            	Test:       	TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification
FAIL
FAIL	github.com/cli/cli/v2/pkg/cmd/attestation/verify	12.350s

macos-latest

https://github.com/cli/cli/actions/runs/12916504935/job/36020835039#step:5:60

--- FAIL: TestVerifyAttestations (1.45s)
    --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (0.37s)
        attestation_integration_test.go:53: 
            	Error Trace:	/Users/runner/work/cli/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:53
            	Error:      	"[0xc000409e[60](https://github.com/cli/cli/actions/runs/12916504935/job/36020835039#step:5:61)]" should have 2 item(s), but has 1
            	Test:       	TestVerifyAttestations/all_attestations_pass_verification
FAIL

ubuntu-latest

https://github.com/cli/cli/actions/runs/12680772503/job/35343196076#step:5:64

--- FAIL: TestVerifyAttestations (1.32s)
    --- FAIL: TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification (0.26s)
        attestation_integration_test.go:[65](https://github.com/cli/cli/actions/runs/12680772503/job/35343196076#step:5:66): 
            	Error Trace:	/home/runner/work/cli/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:65
            	Error:      	"[0xc0009[67](https://github.com/cli/cli/actions/runs/12680772503/job/35343196076#step:5:68)ae0]" should have 2 item(s), but has 1
            	Test:       	TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification
FAIL
FAIL	github.com/cli/cli/v2/pkg/cmd/attestation/verify	6.048s

https://github.com/cli/cli/actions/runs/12659904205/job/35279973709#step:5:64

--- FAIL: TestVerifyAttestations (1.34s)
    --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (0.45s)
        attestation_integration_test.go:53: 
            	Error Trace:	/home/runner/work/cli/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:53
            	Error:      	"[0xc0008511f0]" should have 2 item(s), but has 1
            	Test:       	TestVerifyAttestations/all_attestations_pass_verification
FAIL
FAIL	github.com/cli/cli/v2/pkg/cmd/attestation/verify	6.936s

Logs

Paste the activity from your command line. Redact if needed.

Activity

  1. BagToad commented on Feb 7, 2025

    @BagToad
    Member

    @cli/package-security for help investigating this one 🙏

  2. iamazeem commented on Feb 11, 2025

    @iamazeem
    Contributor

    Getting the same error for Windows:

    --- FAIL: TestVerifyAttestations (1.43s)
        --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (0.41s)
            attestation_integration_test.go:53: 
                	Error Trace:	D:/a/cli/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:53
                	Error:      	"[0xc00090fdf0]" should have 2 item(s), but has 1
                	Test:       	TestVerifyAttestations/all_attestations_pass_verification
    

    Workflow: https://github.com/cli/cli/actions/runs/13251671571/job/36990690943?pr=10334#step:5:61


    Here's another one:

    --- FAIL: TestVerifyAttestations (1.77s)
        --- FAIL: TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification (0.39s)
            attestation_integration_test.go:65: 
                	Error Trace:	D:/a/cli/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:65
                	Error:      	"[0xc000391ef0]" should have 2 item(s), but has 1
                	Test:       	TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification
    

    Workflow: https://github.com/cli/cli/actions/runs/13258224226/job/37008805848?pr=10413#step:5:61


    2025-02-12 (Ubuntu)

    --- FAIL: TestVerifyIntegration (18.63s)
        --- FAIL: TestVerifyIntegration/with_invalid_owner (15.17s)
            verify_integration_test.go:76: 
                	Error Trace:	/home/runner/work/cli/cli/pkg/cmd/attestation/verify/verify_integration_test.go:76
                	Error:      	Error "failed to find recognized issuer from bundle content: failed to create TUF client: tuf refresh failed: context deadline exceeded (Client.Timeout or context cancellation while reading body)" does not contain "expected SourceRepositoryOwnerURI to be https://github.com/fakeowner, got https://github.com/sigstore"
                	Test:       	TestVerifyIntegration/with_invalid_owner
    

    Workflow: https://github.com/cli/cli/actions/runs/13288202067/job/37102016536#step:5:64


    2025-02-13 (macOS)

    WARNING: DATA RACE
    Read at 0x00c000440ae0 by goroutine 72:
      github.com/cli/cli/v2/pkg/cmd/attestation/api.(*failAfterNCallsHttpClient).Get()
          /Users/runner/work/cli/cli/pkg/cmd/attestation/api/mock_httpClient_test.go:69 +0x68
      github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).getBundle.func1()
          /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:194 +0x78
      github.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1()
          /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:18 +0x30
      github.com/cenkalti/backoff/v4.doRetryNotify[go.shape.struct {}]()
          /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:88 +0x15c
      github.com/cenkalti/backoff/v4.RetryNotifyWithTimer()
          /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:61 +0x80
      github.com/cenkalti/backoff/v4.RetryNotify()
          /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:49 +0x1b0
      github.com/cenkalti/backoff/v4.Retry()
          /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:38 +0x190
      github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).getBundle()
          /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:193 +0x138
      github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).fetchBundleFromAttestations.func1()
          /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:169 +0x1f0
      golang.org/x/sync/errgroup.(*Group).Go.func1()
          /Users/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x7c
    
    Previous write at 0x00c000440ae0 by goroutine 73:
      github.com/cli/cli/v2/pkg/cmd/attestation/api.(*failAfterNCallsHttpClient).Get()
          /Users/runner/work/cli/cli/pkg/cmd/attestation/api/mock_httpClient_test.go:69 +0x7c
      github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).getBundle.func1()
          /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:194 +0x78
      github.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1()
          /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:18 +0x30
      github.com/cenkalti/backoff/v4.doRetryNotify[go.shape.struct {}]()
          /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:88 +0x15c
      github.com/cenkalti/backoff/v4.RetryNotifyWithTimer()
          /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:61 +0x80
      github.com/cenkalti/backoff/v4.RetryNotify()
          /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:49 +0x1b0
      github.com/cenkalti/backoff/v4.Retry()
          /Users/runner/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:38 +0x190
      github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).getBundle()
          /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:193 +0x138
      github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).fetchBundleFromAttestations.func1()
          /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:169 +0x1f0
      golang.org/x/sync/errgroup.(*Group).Go.func1()
          /Users/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x7c
    
    Goroutine 72 (running) created at:
      golang.org/x/sync/errgroup.(*Group).Go()
          /Users/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x10c
      github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).fetchBundleFromAttestations()
          /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:1[54](https://github.com/cli/cli/actions/runs/13301543234/job/37143755142?pr=10430#step:5:55) +0xb4
      github.com/cli/cli/v2/pkg/cmd/attestation/api.TestFetchBundleFromAttestations_FailOnTheSecondAttestation()
          /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client_test.go:215 +0x2e0
      testing.tRunner()
          /Users/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1690 +0x184
      testing.(*T).Run.gowrap1()
          /Users/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1743 +0x40
    
    Goroutine 73 (running) created at:
      golang.org/x/sync/errgroup.(*Group).Go()
          /Users/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x10c
      github.com/cli/cli/v2/pkg/cmd/attestation/api.(*LiveClient).fetchBundleFromAttestations()
          /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client.go:154 +0xb4
      github.com/cli/cli/v2/pkg/cmd/attestation/api.TestFetchBundleFromAttestations_FailOnTheSecondAttestation()
          /Users/runner/work/cli/cli/pkg/cmd/attestation/api/client_test.go:215 +0x2e0
      testing.tRunner()
          /Users/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1690 +0x184
      testing.(*T).Run.gowrap1()
          /Users/runner/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1743 +0x40
    ==================
    --- FAIL: TestFetchBundleFromAttestations_FailOnTheSecondAttestation (0.[63](https://github.com/cli/cli/actions/runs/13301543234/job/37143755142?pr=10430#step:5:64)s)
        testing.go:1399: race detected during execution of test
    FAIL
    

    Workflow: https://github.com/cli/cli/actions/runs/13301543234/job/37143755142?pr=10430#step:5:54

  3. malancas commented on Feb 12, 2025

    @malancas
    Contributor

    Thanks for the report, the team is taking a look into this

  4. malancas commented on Feb 13, 2025

    @malancas
    Contributor

    This should be resolved by #10439 but please let us know if you run into this problem again.

  5. iamazeem commented on Feb 14, 2025

    @iamazeem
    Contributor

    @malancas: It's still failing:

    --- FAIL: TestLiveSigstoreVerifier (2.24s)
        --- FAIL: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.42s)
            sigstore_integration_test.go:79: 
                	Error Trace:	D:/a/cli/cli/pkg/cmd/attestation/verification/sigstore_integration_test.go:79
                	Error:      	"[0xc0002c0780]" should have 2 item(s), but has 1
                	Test:       	TestLiveSigstoreVerifier/with_2/3_verified_attestations
    FAIL
    

    Workflow: https://github.com/iamazeem/cli/actions/runs/13334014863/job/37244936343#step:5:60


    2025-02-16 (Windows)

    --- FAIL: TestLiveSigstoreVerifier (1.98s)
        --- FAIL: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.34s)
            sigstore_integration_test.go:79: 
                	Error Trace:	D:/a/cli/cli/pkg/cmd/attestation/verification/sigstore_integration_test.go:79
                	Error:      	"[0xc0004b3430]" should have 2 item(s), but has 1
                	Test:       	TestLiveSigstoreVerifier/with_2/3_verified_attestations
    

    Workflow: https://github.com/cli/cli/actions/runs/13354330624/job/37294953903?pr=10456#step:5:59


    2025-02-18 (Windows)

    --- FAIL: TestLiveSigstoreVerifier (3.02s)
        --- FAIL: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.48s)
            sigstore_integration_test.go:79: 
                	Error Trace:	D:/a/cli/cli/pkg/cmd/attestation/verification/sigstore_integration_test.go:79
                	Error:      	"[0xc00018b8c0]" should have 2 item(s), but has 1
                	Test:       	TestLiveSigstoreVerifier/with_2/3_verified_attestations
    FAIL
    FAIL	github.com/cli/cli/v2/pkg/cmd/attestation/verification	3.302s
    ?   	github.com/cli/cli/v2/pkg/cmd/auth/shared/contract	[no test files]
    --- FAIL: TestVerifyAttestations (1.80s)
        --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (0.61s)
            attestation_integration_test.go:53: 
                	Error Trace:	D:/a/cli/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:53
                	Error:      	"[0xc0009265[60](https://github.com/cli/cli/actions/runs/13383511075/job/37376071790?pr=10456#step:5:61)]" should have 2 item(s), but has 1
                	Test:       	TestVerifyAttestations/all_attestations_pass_verification
    FAIL
    

    Workflow: https://github.com/cli/cli/actions/runs/13383511075/job/37376071790?pr=10456#step:5:59

  6. jsoref commented on Feb 14, 2025

    @jsoref
    ContributorAuthor
  7. iamazeem commented on Feb 17, 2025

    @iamazeem
    Contributor

    UPDATE

    Running tests locally on Windows 10 also reproduces this.
    However, it's a bit random but overall percentage is pretty high i.e. ~80%.

    Command:

    go test -tags=integration .\pkg\cmd\attestation\... -count 2 -failfast
    

    Errors:

    --- FAIL: TestVerifyAttestations (1.13s)
        --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (1.12s)
            attestation_integration_test.go:53:
                    Error Trace:    C:/cli-trunk/pkg/cmd/attestation/verify/attestation_integration_test.go:53
                    Error:          "[0xc000719230]" should have 2 item(s), but has 1
                    Test:           TestVerifyAttestations/all_attestations_pass_verification
    
    --- FAIL: TestLiveSigstoreVerifier (3.09s)
        sigstore_integration_test.go:62:
                    Error Trace:    C:/cli-trunk/pkg/cmd/attestation/verification/sigstore_integration_test.go:62
                    Error:          Not equal:
                                    expected: 2
                                    actual  : 1
                    Test:           TestLiveSigstoreVerifier
                    Messages:       test case: with valid artifact and JSON lines file containing multiple Sigstore bundles
    
    --- FAIL: TestVerifyAttestations (3.72s)
        --- FAIL: TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification (1.42s)
            attestation_integration_test.go:65:
                    Error Trace:    C:/cli-trunk/pkg/cmd/attestation/verify/attestation_integration_test.go:65
                    Error:          "[0xc000781d30]" should have 2 item(s), but has 1
                    Test:           TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification
    
    --- FAIL: TestVerifyAttestations (1.85s)
        --- FAIL: TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification (0.18s)
            attestation_integration_test.go:63:
                    Error Trace:    C:/cli-trunk/pkg/cmd/attestation/verify/attestation_integration_test.go:63
                    Error:          Received unexpected error:
                                    failed to get bundle issuer: unsupported bundle version: application/vnd.dev.sigstore.bundle+json;version=0.1
                    Test:           TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification
    
    --- FAIL: TestLiveSigstoreVerifier (3.61s)
        --- FAIL: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.82s)
            sigstore_integration_test.go:79:
                    Error Trace:    C:/cli-trunk/pkg/cmd/attestation/verification/sigstore_integration_test.go:79
                    Error:          "[0xc000293310]" should have 2 item(s), but has 1
                    Test:           TestLiveSigstoreVerifier/with_2/3_verified_attestations
    

    Following two JSON files are generated by running these tests:

    1. pkg/cmd/attestation/verification/tuf-repo.github.com.json
    2. pkg/cmd/attestation/verification/tuf-repo-cdn.sigstore.dev.json

    and are not removed automatically.
    Looks like they may be removed when the tests finish and/or added to .gitignore.

  8. williammartin commented on Feb 18, 2025

    @williammartin
    Member

    @iamazeem are you running in WSL? Powershell? I can't seem to reproduce this on my Parallels Windows 11 Enterprise.

    For the debris files I created #10462

  9. iamazeem commented on Feb 18, 2025

    @iamazeem
    Contributor

    @williammartin: I'm running these tests in Powershell on Windows 10 Home.
    Besides, these tests are failing in CI pipeline also as mentioned above.

  10. williammartin commented on Feb 18, 2025

    @williammartin
    Member

    Besides, these tests are failing in CI pipeline also as mentioned #10390 (comment).

    Sure, I'm just trying to reproduce it myself to think about fixing it. Thanks.

  11. added
    tech-debtA chore that addresses technical debt
    and removed
    bugSomething isn't working
    on Feb 19, 2025
  12. williammartin commented on Feb 20, 2025

    @williammartin
    Member

    FWIW this branch contains some modifications that I believe only run 2 tests, and it fails very frequently (after I bumped the specs on my Windows VM):

    PS C:\Users\williammartin\workspace\cli> go test -v -tags=integration ./pkg/cmd/attestation/verification ./pkg/cmd/attestation/verify -count 2 -failfast
    === RUN   TestLiveSigstoreVerifier
    === RUN   TestLiveSigstoreVerifier/with_2/3_verified_attestations
    --- PASS: TestLiveSigstoreVerifier (0.78s)
        --- PASS: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.78s)
    === RUN   TestLiveSigstoreVerifier
    === RUN   TestLiveSigstoreVerifier/with_2/3_verified_attestations
    --- PASS: TestLiveSigstoreVerifier (0.59s)
        --- PASS: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.59s)
    PASS
    ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  1.499s
    === RUN   TestVerifyAttestations
    === RUN   TestVerifyAttestations/all_attestations_pass_verification
        attestation_integration_test.go:53:
                    Error Trace:    C:/Users/williammartin/workspace/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:53
                    Error:          "[0x400045ae30]" should have 2 item(s), but has 1
                    Test:           TestVerifyAttestations/all_attestations_pass_verification
    --- FAIL: TestVerifyAttestations (0.43s)
        --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (0.43s)
    FAIL
    FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verify        0.548s
    FAIL
    
  13. williammartin commented on Feb 20, 2025

    @williammartin
    Member

    I believe this is occurring because instantiation of the TUF Client results in a json file being written to disk via a rename operation, and this is not concurrent safe on Windows: https://github.com/theupdateframework/go-tuf/blob/f59c91f01f8a64466d0ecbe0cd303469615de1e9/metadata/updater/updater.go#L628

    Here's the error I eventually managed to get out of the tests:

    sigstore.go:227: FAIL: verifying: failed to find recognized issuer from bundle content: failed to create TUF client: tuf refresh failed: rename C:\Users\williammartin\AppData\Local\GitHub CLI\.sigstore\root\tuf-repo-cdn.sigstore.dev\tuf_tmp3512682291 C:\Users\williammartin\AppData\Local\GitHub CLI\.sigstore\root\tuf-repo-cdn.sigstore.dev\root.json: Access is denied.
    

    From the windows godoc:

    // Rename renames (moves) oldpath to newpath.
    // If newpath already exists and is not a directory, Rename replaces it.
    // OS-specific restrictions may apply when oldpath and newpath are in different directories.
    // Even within the same directory, on non-Unix platforms Rename is not an atomic operation. <--------
    // If there is an error, it will be of type *LinkError.
    
  14. williammartin commented on Feb 20, 2025

    @williammartin
    Member

    Some further validation, when I run the tests with CODESPACES=true, I cannot reproduce this flake. This makes sense because this env var results in us setting a TUF client config option that results in the file writing being skipped.

  15. williammartin commented on Feb 21, 2025

    @williammartin
    Member

    I think #10478 should resolve this but will reopen if we see these flakes reappear in CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

tech-debtA chore that addresses technical debt

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions