Summary
A BucketClaim can stay status.bucketReady=false indefinitely even though its backend Bucket reports status.bucketReady=true. Because the readiness never propagates, the dependent BucketAccess is never granted and credentials are never issued, so the consuming app never gets a working bucket. It is intermittent (flapping), not a global outage — separate bucket suites can pass in the same run.
Reproductions
I have seen this surface independently in two unrelated E2E runs:
- harbor E2E: the
harbor-test-registry BucketClaim never reached bucketReady within 10 minutes, while the backend Bucket was already READY=true.
- bucket E2E: a
bucket-test BucketClaim stayed READY=false while the backend Bucket was READY=true; the downstream bucketaccess/bucket-test-admin then failed with WaitingForBucket: invalid bucket state: BucketAccess can't be granted to Bucket not in Ready state.
The repo already encodes awareness of this lag: hack/e2e-apps/harbor.bats comments that "the central COSI controller's propagation of the Bucket's readiness back onto the namespaced BucketClaim can lag several minutes" and waits up to 600s, while hack/e2e-apps/bucket.bats waits 300s.
Mechanism (confirmed)
The central object-storage controller is built from upstream COSI v0.2.2 (packages/system/objectstorage-controller/images/objectstorage/Dockerfile). In that release the BucketClaim reconcile (controller/pkg/bucketclaim/bucketclaim.go, provisionBucketClaimOperation) hardcodes bucketReady=false in the provisioning path and returns without requeueing, and the controller registers no Bucket→BucketClaim propagation handler — the Bucket readiness transition is performed by the sidecar on a different object.
There is already a downstream fix for this, 92-bucketclaim-propagate-ready.diff, which re-reads the live Bucket after create and copies its bucketReady onto the BucketClaim. This patch is present in the tree that both reproductions were built from, yet the desync still occurs — so the patch is incomplete.
The reason it is incomplete: the patch relies on the controller's periodic 30s informer resync to re-run the reconcile and converge. But the COSI generic controller (internal/runtime/controller.go) drops no-op resync deltas (if reflect.DeepEqual(d.Object, old) { return nil }). After the first reconcile nothing ever mutates the BucketClaim — the sidecar only mutates the Bucket — so the BucketClaim never produces a changed delta, the resync never re-invokes the listener, and the re-read never happens again.
Net effect: the BucketClaim's readiness is frozen at whatever value it had during its first reconcile. The only way it becomes true is if the backend Bucket was already ready at that exact instant, which is a race between the async sidecar (backend provisioning) and the in-process controller read-back. The sidecar usually loses that race, so the claim is stuck false; occasionally it wins, so the claim is true. That is the flapping.
Hypothesis (not yet logged)
I inferred the per-failure ordering (the controller read-back saw bucketReady=false because the sidecar had not yet flipped it) from the mechanism rather than from the failing runs' controller logs; capturing objectstorage-controller logs at --v=5 during a failure would confirm it directly. The reflect.DeepEqual resync-drop should also be eyeballed against the upstream source to confirm verbatim.
Possible directions (not prescriptive)
Make the controller actually react to the Bucket readiness transition rather than rely on a resync that never fires — e.g. a controller-side Bucket informer/handler that enqueues the owning BucketClaim, an explicit requeue while bucketReady=false, or having the sidecar also stamp the BucketClaim. The downstream patch commit notes upstream's rewritten controller (main) already handles this, so bumping/rebasing onto the rewritten controller is another option.
Related
Distinct from #2429 (provisioner CrashLoop from a stale unix socket); same area, different failure mode.
Summary
A
BucketClaimcan staystatus.bucketReady=falseindefinitely even though its backendBucketreportsstatus.bucketReady=true. Because the readiness never propagates, the dependentBucketAccessis never granted and credentials are never issued, so the consuming app never gets a working bucket. It is intermittent (flapping), not a global outage — separate bucket suites can pass in the same run.Reproductions
I have seen this surface independently in two unrelated E2E runs:
harbor-test-registryBucketClaim never reachedbucketReadywithin 10 minutes, while the backend Bucket was alreadyREADY=true.bucket-testBucketClaim stayedREADY=falsewhile the backend Bucket wasREADY=true; the downstreambucketaccess/bucket-test-adminthen failed withWaitingForBucket: invalid bucket state: BucketAccess can't be granted to Bucket not in Ready state.The repo already encodes awareness of this lag:
hack/e2e-apps/harbor.batscomments that "the central COSI controller's propagation of the Bucket's readiness back onto the namespaced BucketClaim can lag several minutes" and waits up to 600s, whilehack/e2e-apps/bucket.batswaits 300s.Mechanism (confirmed)
The central object-storage controller is built from upstream COSI v0.2.2 (
packages/system/objectstorage-controller/images/objectstorage/Dockerfile). In that release the BucketClaim reconcile (controller/pkg/bucketclaim/bucketclaim.go,provisionBucketClaimOperation) hardcodesbucketReady=falsein the provisioning path and returns without requeueing, and the controller registers no Bucket→BucketClaim propagation handler — the Bucket readiness transition is performed by the sidecar on a different object.There is already a downstream fix for this,
92-bucketclaim-propagate-ready.diff, which re-reads the live Bucket after create and copies itsbucketReadyonto the BucketClaim. This patch is present in the tree that both reproductions were built from, yet the desync still occurs — so the patch is incomplete.The reason it is incomplete: the patch relies on the controller's periodic 30s informer resync to re-run the reconcile and converge. But the COSI generic controller (
internal/runtime/controller.go) drops no-op resync deltas (if reflect.DeepEqual(d.Object, old) { return nil }). After the first reconcile nothing ever mutates the BucketClaim — the sidecar only mutates the Bucket — so the BucketClaim never produces a changed delta, the resync never re-invokes the listener, and the re-read never happens again.Net effect: the BucketClaim's readiness is frozen at whatever value it had during its first reconcile. The only way it becomes
trueis if the backend Bucket was already ready at that exact instant, which is a race between the async sidecar (backend provisioning) and the in-process controller read-back. The sidecar usually loses that race, so the claim is stuckfalse; occasionally it wins, so the claim istrue. That is the flapping.Hypothesis (not yet logged)
I inferred the per-failure ordering (the controller read-back saw
bucketReady=falsebecause the sidecar had not yet flipped it) from the mechanism rather than from the failing runs' controller logs; capturing objectstorage-controller logs at--v=5during a failure would confirm it directly. Thereflect.DeepEqualresync-drop should also be eyeballed against the upstream source to confirm verbatim.Possible directions (not prescriptive)
Make the controller actually react to the Bucket readiness transition rather than rely on a resync that never fires — e.g. a controller-side Bucket informer/handler that enqueues the owning BucketClaim, an explicit requeue while
bucketReady=false, or having the sidecar also stamp the BucketClaim. The downstream patch commit notes upstream's rewritten controller (main) already handles this, so bumping/rebasing onto the rewritten controller is another option.Related
Distinct from #2429 (provisioner CrashLoop from a stale unix socket); same area, different failure mode.