seaweedfs: e2e sandbox runs out of volume slots, so late backup suites cannot write to a fresh bucket #4239

Describe the bug

Late in a full e2e run, S3 writes stop working: the sandbox's SeaweedFS has no volume slots left, so the master cannot grow a volume for a newly created bucket and every PUT through the S3 gateway times out.

It surfaces as a backup suite failing on its access preflight, which reads like a credentials or COSI problem and is not one. The COSI objects are fine:

BucketAccess tenant-root/bucket-clickhouse-backups-backup
  status: {accessGranted: true, accountID: ba-93e6bf6f-...}
BucketClaim  tenant-root/bucket-clickhouse-backups
  status: {bucketName: bucket-6af22edd-..., bucketReady: true}

Every SeaweedFS pod is Running and ready with zero restarts, which is why this looks like a flake rather than a resource ceiling.

Environment

• Cozystack: main
• In-tree e2e lane, container sandbox
• Seen in run 34864316378, suite clickhouse-2-backup-roundtrip

To Reproduce

Run the full chainsaw suite. The failure lands on whichever backup suite runs late enough; each suite creates its own buckets, and every bucket needs volumes.

Expected behaviour

A bucket created part-way through a run is writable.

Actual behaviour

The chain, top to bottom.

The preflight gives up after 90s:

BucketAccess tenant-root/bucket-clickhouse-backups-backup was granted but did not pass S3 PUT/GET/DELETE within 90s

The S3 gateway cannot place the object, three attempts 20s apart:

E0914 18:27:42 s3api_object_handlers_put.go:526 putToFiler: chunked upload failed:
  assign volume: all filers failed, last error:
  rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0914 18:28:02 ... same
E0914 18:28:22 ... stream terminated by RST_STREAM with error code: CANCEL

Both filers retry assign against the master and give up:

I0914 18:28:22 grpc_client_server.go:427 Removing cached gRPC connection to
  seaweedfs-master-0.seaweedfs-master.tenant-root:19333 due to error: DeadlineExceeded
I0914 18:28:22 retry.go:124 retry assign: retrying in 1s: DeadlineExceeded

And the master says why:

I0914 18:28:23 master_grpc_server_volume.go:151 volume grow Option:{"collection":"bucket-6af22edd-...","replication":{"node":1}} Reason:grpc assign
I0914 18:28:23 volume_growth.go:135 create 6 volume, created 0: Not enough data nodes found!
I0914 18:28:23 master_grpc_server_volume.go:42 automatic volume grow failed: Not enough data nodes found!

Not enough data nodes found appears 2371 times in one master's log for this run, so this is a settled state rather than a blip.

Additional context

The arithmetic behind "not enough data nodes", from the running pods:

With a 1000MB volume size and ~9.9GB free, the auto limit lands around 9-10 volumes per server, and replication 001 spends a slot on each server per volume. A full run creates buckets suite after suite, so by the time a late backup suite asks for one, there is nothing left to allocate — while the disk still looks 99% free, because the ceiling is the slot count and not the bytes.

That also explains the intermittency: the same suite passes when it runs earlier, and the failure moves between suites depending on ordering. In an earlier run clickhouse-2-backup-roundtrip passed and a different suite failed.

Worth deciding separately from this report: whether the sandbox should raise the per-server volume ceiling, grow the PVCs, drop defaultReplication to 000 for a two-server test rig, or reclaim buckets between suites. Each trades something different, and the choice is a test-infrastructure call rather than a mechanical one.

Checklist

• I have checked the documentation
• I have searched for similar issues
• I have included all required information
• I have provided clear steps to reproduce
• I have included relevant logs