test(e2e): wait for the S3 gateway to accept a key before using it - #4694

What this PR does

The bucket e2e suite fails now and then because the S3 gateway rejects a key that COSI already reports as granted. The error is The access key ID you provided does not exist in our records. from mc alias set. In job 109130131391 the BucketAccess assert finished at 21:23:43.57 and the first mc alias set failed at 21:23:44.82. In job 109396174094 cap1 and cap2 were accepted, and cap3 was rejected two seconds into the step.

Granted access is not the same as a usable key. In seaweedfs-cosi-driver v0.3.1, DriverGrantBucketAccess writes the new key into /etc/iam/identity.json on the filer and returns. The S3 gateway in SeaweedFS 4.45 reloads that file later, from a filer metadata subscription (weed/s3api/auth_credentials_subscribe.go). In our chart the driver writes through seaweedfs-filer and the two gateways follow seaweedfs-filer-client, so the change may have to pass through the other filer replica first.

The suite now waits until the gateway accepts the key before it writes. Every alias in both write steps goes through await_key, which uses the same port-forward as the writes and so asks the same gateway pod. It waits only while the answer is the InvalidAccessKeyId text, for at most 60s. Any other error fails at once, and so does an attempt cut off at its 15s cap. When the window runs out, the step fails with a message that names the BucketAccess.

One probe costs at most 77s, so both steps go from a 5m to an 8m op timeout. The inner budgets now sum to 409s and 441s, and the budget comments say so.

I checked the probe locally against a fake mc:

case=ok rc=0 attempts=1 elapsed=0s
  Added `a` successfully.
case=lag rc=0 attempts=3 elapsed=5s
  Added `a` successfully.
case=other rc=1 attempts=1 elapsed=0s
  FAIL: mc alias set for BucketAccess bucket-test-admin failed (exit 1) with an error other than the key not being loaded yet
case=never rc=1 attempts=31 elapsed=60s
  FAIL: S3 gateway still rejects the key of BucketAccess bucket-test-admin after 60s of probing (the gateway never loaded it, or the driver never wrote it)

A hanging mc fails on its first attempt after 15s with exit 124. chainsaw lint test from v0.2.15 accepts the file.

The read-only upload check gets fixed along the way. Before, an unloaded ro-user key also made the upload fail, and that failure passed as the expected denial.

One thing I left as it is: await_key is copied into both steps, because chainsaw steps share no shell, and moving it to _lib/ would run every suite on each edit.

Closes #4580

Screenshots

Not applicable, no UI change.

Downstream repositories

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:
• cozystack/community - follow-up:

Release note

test(e2e): the bucket suite waits for the S3 gateway to accept a new key before writing with it, instead of trusting BucketAccess readiness.