Skip to content

test(e2e): bucket suite writes before SeaweedFS accepts a key that BucketAccess already reports Ready #4580

Closed

Description

The bucket Chainsaw suite fails now and then with mc alias set rejecting a freshly issued key: "The access key ID you provided does not exist in our records". The BucketAccess for that key is already Ready and its Secret is non-empty, so the key exists on the COSI side while the SeaweedFS S3 gateway does not accept it yet.

Two runs so far, on PRs whose diffs do not touch seaweedfs, COSI, the bucket app or the tests:

Both steps are in hack/e2e-chainsaw/bucket/chainsaw-test.yaml (mc alias set at lines 151-152 and 389 on main). The test file has not changed since ac92b6b (2026-09-22).

The mechanism is my reading of the timings. I did not trace it in the seaweedfs or cosi-driver source. Ready on the BucketAccess says the credentials were issued and written to the Secret. It does not say the S3 gateway has loaded them, so anything that uses the key right after Ready can race.

A fix in the test would wait until the gateway accepts the key, for example by retrying mc alias set (or mc ls on the bucket) for a bounded time before the write, in both steps. If the driver is meant to report Ready only once the key is usable, that part belongs in the driver instead.

Activity

  1. added
    kind/flakeCategorizes issue or PR as related to a flaky test
    area/testingIssues or PRs related to testing (e2e, bats, unit tests)
    on Sep 29, 2026
  2. added a commit that references this issue on Oct 5, 2026
    1ca603a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/flakeCategorizes issue or PR as related to a flaky testtriage/needs-triageIndicates an issue needs triage by a maintainer

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions