objectstorage-controller: merged BucketClaim requeue fix (#3034) is not deployed — pinned digest predates it (harbor E2E flake + systemic gap) #3143

Summary

A fix that is already merged on main is not running anywhere, because the deployed image digest predates it. Concretely: PR #3034 added a "requeue the BucketClaim until the backend Bucket is ready" loop to objectstorage-controller, but the controller image pinned in packages/system/objectstorage-controller/values.yaml was last rebuilt at the v1.5.0 release-prep, three days before #3034 merged. So the running controller still has the old behaviour, and the harbor E2E BucketClaim readiness check flakes.

This is two problems in one: (1) the concrete harbor flake, and (2) a systemic gap — any first-party image-source fix merged to main stays inert in E2E and in production until the next release-prep repins its digest.

The concrete bite: harbor E2E BucketClaim never reaches bucketReady

The harbor E2E test waits for BucketClaim harbor-test-registry to reach status.bucketReady: true. It intermittently times out: the backend Bucket is provisioned fine, but the BucketClaim is stuck false forever.

Evidence (E2E run on PR #3132)

Observed in a recent E2E run — the run on PR #3132, whose own changes are unrelated to object storage; it is just where a fresh cozyreport happened to capture the failure. In the cozyreport bundle from that run:

• Backend Bucket harbor-test-registry → status.bucketReady: true (provisioned and access-granted by the sidecar).
• BucketClaim harbor-test-registry → status.bucketReady: false, and it stays false for the ~39 minutes the log covers.
• Two sibling claims in the same run (bucket-test, bucket-cozy-backups) converged to true — so the SeaweedFS provisioner, filer and S3 path are healthy. This is not a storage-side or all-in-one-wiring problem.
• The central controller reconciled the harbor claim only twice, both in the first ~0.8s (14:48:15.565 and 16.318), then never again. The backend Bucket flipped ready at ~14:48:15.65 — after the controller's last reconcile of the claim.

Why the running binary is missing the fix

The deployed objectstorage-controller logs Finished creating Bucket at bucketclaim.go:300. With #3034's full patch applied, that same log call sits at bucketclaim.go:316, and the requeue guard (backend Bucket ... not ready yet; requeuing, event reason WaitingForBucket) is at lines 312–313. The 16-line offset shows the running binary contains the earlier readiness-propagation change (PR #2792) but not #3034's requeue. Consistently, the requeue/WaitingForBucket message never appears anywhere in the controller log.

The deployed image is objectstorage-controller:v1.5.0@sha256:e27da946… — byte-for-byte the digest pinned in values.yaml. git log shows that line is only ever changed by Prepare release commits; the current digest was set by Prepare release v1.5.0 (2026-06-22). #3034 merged 2026-06-25 and did not touch values.yaml or rebuild the image. The per-package Build job for objectstorage-controller was skipped in #3132's run (its source was untouched), so E2E pulled the pinned, pre-#3034 image.

Why it is an intermittent race, not a hard break

Without the requeue, the controller samples the backend Bucket's readiness once per reconcile event and never re-drives the claim afterwards: it does not watch Bucket objects, and the 30s informer resync is dropped as a no-op delta. If the sidecar flips Bucket.status.bucketReady to true after the controller's last reconcile of the claim, the claim is stuck false permanently.

Whether it bites is sub-second timing. bucket-test won the race — its second reconcile ran ~350ms after the first (after the readiness flip), saw the claim already true, and short-circuited. The harbor claim lost — its second reconcile started ~0.5ms after the first and sampled the Bucket before the flip, wrote the stale false, and was never reconciled again. #3034's requeue closes exactly this gap.

The systemic gap

E2E and production run first-party images by their pinned digests. A PR's Build job rebuilds only the images whose source changed in that PR, patches the new digest into values.yaml, and feeds that patch to E2E. Unchanged images keep the committed digest. The committed digests on main are only refreshed at release-prep.

The consequence: a fix merged into a first-party image's source is exercised by its own PR's E2E (which rebuilt the image), then goes inert after merge until the next release repins the digest. Between releases, main runs the previous release's binaries for every package no one happened to re-touch. #3034 is one instance; any image-source fix merged since v1.5.0 is in the same state, silently not in effect in E2E or production.

This also makes such fixes hard to notice as un-shipped: the source has them, their unit tests pass, and the PR that introduced them went green — the only place the old behaviour shows is a run that did not rebuild that particular image.

Options to discuss

This touches the release/digest model, so I am listing options rather than proposing one.

• A — do nothing extra. fix(objectstorage-controller): converge BucketClaim readiness and speed up COSI provisioner failover #3034's requeue is in main's source and will ship at the next release-prep when all digests are rebuilt. The harbor flake persists on E2E (and in v1.5.0) until then.
• B — fast-track a manual repin now. Rebuild objectstorage-controller (and the objectstorage-sidecar it shares a build with) from current main and commit the new digests in packages/system/objectstorage-controller/values.yaml and packages/extra/seaweedfs/values.yaml (plus the objectstorage-sidecar.tag). Ships fix(objectstorage-controller): converge BucketClaim readiness and speed up COSI provisioner failover #3034 immediately and stops the flake, but is an out-of-band digest change in a flow where digests are otherwise release-managed.
• C — repin changed images automatically on merge. Have CI rebuild and repin the digest for any first-party image whose source changed when a PR lands on main, so a merged image-source fix takes effect immediately. Tradeoff: digests would no longer be strictly release-managed, which interacts with the immutable-release-tag direction in RFC: immutable release tags & rc-promotion flow #2677 and the build-pipeline redesign in Finish the CI image-build refactor: SBOM emission and pr-* tag retention #2937 (where the per-package yq/pr.patch digest flow already lives).

Related

• PR fix(objectstorage-controller): converge BucketClaim readiness and speed up COSI provisioner failover #3034 — the requeue fix that is merged but not deployed.
• PR fix(objectstorage-controller): propagate Bucket readiness to BucketClaim #2792 — the readiness-propagation change that the deployed binary does have (and which is not sufficient on its own).
• Finish the CI image-build refactor: SBOM emission and pr-* tag retention #2937 — CI image-build refactor; owns the per-package digest flow that option C would extend.
• RFC: immutable release tags & rc-promotion flow #2677 — immutable release tags / rc-promotion; the release-model context for any change to how digests are pinned.