Skip to content

Replace system-namespace projection tick with a watch on the source Secret #2735

Open

Description

Background

Follow-up to #2716.

SystemCredentialsProjector (internal/backupcontroller/credentials_projector_runnable.go) re-projects cozy-backups-creds into the system namespaces (cozy-velero, plus anything in backupStorage.systemNamespaces) on a fixed 1-minute time.Ticker. It is leader-elected so only one controller-manager replica ticks. The tick exists because those namespaces hold long-lived static references (Velero's BackupStorageLocation) that must have valid credentials at all times, with no per-Job event to hang projection off.

Proposal

Convert the timer-driven Runnable into a controller-runtime reconciler that watches the source Secret (cozy-backup-controller/<source>) and re-projects on change, instead of polling every 60s. Benefits:

  • Steady-state apiserver load drops to ~zero (the source Secret rarely changes), instead of N projections/minute forever.
  • Credential rotation propagates immediately rather than within up-to-60s.
  • Removes the bespoke ticker loop in favour of standard reconcile/requeue/backoff semantics.

Leader-election is unchanged — a standard controller is leader-elected by default, preserving the single-projector-per-cluster property documented at credentials_projector_runnable.go:70-75.

Caveat — keep the self-healing the tick currently provides

The 1-minute tick does double duty: besides propagating source changes, it self-heals target drift. If someone deletes or edits the projected Secret in cozy-velero, the next tick recreates it within 60s — this is what keeps the Velero BSL from sticking on Unavailable (see docs/operations/backup-classes.md:88-92). A pure source-Secret watch would miss target deletion.

So the reconciler should also:

  • Watch (or Owns) the target Secrets and re-project on target change/deletion, and/or
  • Keep a long resync backstop (manager SyncPeriod, or a periodic RequeueAfter on the order of 10–30m) instead of the current 60s hot loop — long enough to stop being a busy-poll, short enough to bound drift if a watch event is ever missed.

Acceptance

  • No periodic projection when nothing changes (verify via the cozystack_backup_credentials_projection_successes_total counter staying flat at idle).
  • Source-Secret rotation and target-Secret deletion both trigger a re-projection promptly (e2e).
  • Velero BSL recovers to Available after a manual delete of the projected Secret in cozy-velero without waiting on a slow backstop.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area/storageIssues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor)kind/cleanupCategorizes issue or PR as related to cleanup of code, process, or technical debtpriority/backlogGeneral backlog priority. Lower than priority/important-longtermtriage/acceptedIndicates an issue is ready to be actively worked on

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions