Skip to content

Helm lookup-gated backup objects are skipped once and never created, which makes migration 50 fail-closed #3518

Description

@mattia-eleuteri

What happened

Several objects in packages/system/backupstrategy-controller and packages/system/bucket are gated on a Helm
lookup of an object that the same release (or a sibling release) is still creating. When the lookup comes back
empty the object is silently skipped — and because helm-controller does not re-render without a chart or values
change (drift detection is off by default), it is never created at all.

On our cluster, months after install, the following were all missing:

  • the 7 default backup Strategy CRs (cozy-default-etcd, -cnpg, -mariadb, -altinity, -foundationdb,
    -velero-vmdisk, -velero-vminstance), gated on {{- if $bucketName -}} where bucketName comes from a
    lookup of BucketClaim;
  • the Velero BackupStorageLocation (velero-bsl.yaml), same gate;
  • the <bucket>-<user>-credentials Secret (packages/system/bucket/templates/user-credentials.yaml), gated on a
    lookup of the COSI Secret.

Only BackupClass cozy-default existed, because its template is rendered unconditionally — its own comment even
predicts this situation and assumes "Flux re-renders the chart", which does not happen.

Why it matters

Migration 50 (etcd adoption, v1.6.0) is fail-closed on the snapshot target and reboots its Job forever:

if the target cannot be resolved we FAIL LOUDLY rather than adopt without a snapshot

So a cluster that hit this race cannot be upgraded to v1.6.0 at all, and the documented escape hatch
(migrations.etcdAdoptSkipBackup) means adopting a live etcd — potentially a Kamaji datastore — with no snapshot.

The install timing that caused it was a 4-second window: backupstrategy-controller deployed at 14:10:24, the
cozy-backups Bucket at 14:10:28.

Recovering required forcing a real Helm upgrade (reconcile.fluxcd.io/forceAt and requestedAt; a plain
reconcile does nothing) on two releases, in order, because the race is nested: first
backupstrategy-controller, then bucket-<name>-system for the credentials Secret. That second one is easy to
miss: the Secret is rendered by the -system release, not by bucket-<name>.

Suggested fix

Do not gate object creation on a render-time lookup. Either have backupstrategy-controller reconcile these
objects itself (it is a controller and already watches the relevant types), or make the Bucket provisioning a
dependsOn so ordering is guaranteed instead of hoped for.

If the templating approach is kept, the chart should at least fail loudly (fail) rather than silently emit
nothing, so the condition is visible at install time instead of years later during an upgrade.

Environment

Cozystack v1.5.3 → v1.6.0, backupStorage.provisionBucket: true, SeaweedFS-backed platform bucket.

Activity

  1. added a commit that references this issue on Aug 10, 2026
    cfaf2e0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions