What happened
Several objects in packages/system/backupstrategy-controller and packages/system/bucket are gated on a Helm
lookup of an object that the same release (or a sibling release) is still creating. When the lookup comes back
empty the object is silently skipped — and because helm-controller does not re-render without a chart or values
change (drift detection is off by default), it is never created at all.
On our cluster, months after install, the following were all missing:
- the 7 default backup
Strategy CRs (cozy-default-etcd, -cnpg, -mariadb, -altinity, -foundationdb,
-velero-vmdisk, -velero-vminstance), gated on {{- if $bucketName -}} where bucketName comes from a
lookup of BucketClaim;
- the Velero
BackupStorageLocation (velero-bsl.yaml), same gate;
- the
<bucket>-<user>-credentials Secret (packages/system/bucket/templates/user-credentials.yaml), gated on a
lookup of the COSI Secret.
Only BackupClass cozy-default existed, because its template is rendered unconditionally — its own comment even
predicts this situation and assumes "Flux re-renders the chart", which does not happen.
Why it matters
Migration 50 (etcd adoption, v1.6.0) is fail-closed on the snapshot target and reboots its Job forever:
if the target cannot be resolved we FAIL LOUDLY rather than adopt without a snapshot
So a cluster that hit this race cannot be upgraded to v1.6.0 at all, and the documented escape hatch
(migrations.etcdAdoptSkipBackup) means adopting a live etcd — potentially a Kamaji datastore — with no snapshot.
The install timing that caused it was a 4-second window: backupstrategy-controller deployed at 14:10:24, the
cozy-backups Bucket at 14:10:28.
Recovering required forcing a real Helm upgrade (reconcile.fluxcd.io/forceAt and requestedAt; a plain
reconcile does nothing) on two releases, in order, because the race is nested: first
backupstrategy-controller, then bucket-<name>-system for the credentials Secret. That second one is easy to
miss: the Secret is rendered by the -system release, not by bucket-<name>.
Suggested fix
Do not gate object creation on a render-time lookup. Either have backupstrategy-controller reconcile these
objects itself (it is a controller and already watches the relevant types), or make the Bucket provisioning a
dependsOn so ordering is guaranteed instead of hoped for.
If the templating approach is kept, the chart should at least fail loudly (fail) rather than silently emit
nothing, so the condition is visible at install time instead of years later during an upgrade.
Environment
Cozystack v1.5.3 → v1.6.0, backupStorage.provisionBucket: true, SeaweedFS-backed platform bucket.
What happened
Several objects in
packages/system/backupstrategy-controllerandpackages/system/bucketare gated on a Helmlookupof an object that the same release (or a sibling release) is still creating. When the lookup comes backempty the object is silently skipped — and because helm-controller does not re-render without a chart or values
change (drift detection is off by default), it is never created at all.
On our cluster, months after install, the following were all missing:
StrategyCRs (cozy-default-etcd,-cnpg,-mariadb,-altinity,-foundationdb,-velero-vmdisk,-velero-vminstance), gated on{{- if $bucketName -}}wherebucketNamecomes from alookupofBucketClaim;BackupStorageLocation(velero-bsl.yaml), same gate;<bucket>-<user>-credentialsSecret (packages/system/bucket/templates/user-credentials.yaml), gated on alookupof the COSI Secret.Only
BackupClass cozy-defaultexisted, because its template is rendered unconditionally — its own comment evenpredicts this situation and assumes "Flux re-renders the chart", which does not happen.
Why it matters
Migration 50 (etcd adoption, v1.6.0) is fail-closed on the snapshot target and reboots its Job forever:
So a cluster that hit this race cannot be upgraded to v1.6.0 at all, and the documented escape hatch
(
migrations.etcdAdoptSkipBackup) means adopting a live etcd — potentially a Kamaji datastore — with no snapshot.The install timing that caused it was a 4-second window:
backupstrategy-controllerdeployed at14:10:24, thecozy-backupsBucket at14:10:28.Recovering required forcing a real Helm upgrade (
reconcile.fluxcd.io/forceAtandrequestedAt; a plainreconcile does nothing) on two releases, in order, because the race is nested: first
backupstrategy-controller, thenbucket-<name>-systemfor the credentials Secret. That second one is easy tomiss: the Secret is rendered by the
-systemrelease, not bybucket-<name>.Suggested fix
Do not gate object creation on a render-time
lookup. Either havebackupstrategy-controllerreconcile theseobjects itself (it is a controller and already watches the relevant types), or make the Bucket provisioning a
dependsOnso ordering is guaranteed instead of hoped for.If the templating approach is kept, the chart should at least fail loudly (
fail) rather than silently emitnothing, so the condition is visible at install time instead of years later during an upgrade.
Environment
Cozystack v1.5.3 → v1.6.0,
backupStorage.provisionBucket: true, SeaweedFS-backed platform bucket.