Repository navigation
Garbage-collect projected cozy-backups-creds Secrets via ownerReferences #2734
Description
Background
Follow-up to #2716.
The credentials projector (internal/backupcontroller/credentials_projector.go) copies the platform-managed S3 Secret into a tenant namespace as cozy-backups-creds, on demand right before each BackupJob/RestoreJob runs (backupjob_controller.go:150, restorejob_controller.go:136). The projected Secret carries the apps.cozystack.io/managed-by label but no ownerReferences. As a result, once a tenant runs its last backup the projected Secret lingers in the tenant namespace indefinitely, holding usable platform S3 credentials with no lifecycle tied to anything.
Proposal
Set ownerReferences on the projected Secret so the Kubernetes garbage collector reclaims it once its owner is gone, shrinking the window in which platform credentials sit idle in tenant namespaces. This is complementary to — not a replacement for — the existing managed-by ownership guard (credentials_projector.go:205-213): the label prevents clobbering a Secret the projector doesn't own; ownerReferences drive cleanup.
Design constraint — do NOT owner-ref the triggering BackupJob
The obvious "GC when the backup is done" (owner = the BackupJob) is unsafe, because the single cozy-backups-creds Secret is shared by consumers that outlive a one-shot backup:
- ClickHouse
useSystemBucket— theclickhouse-backupsidecar runs inside the long-lived ClickHouse Pod and mountscozy-backups-creds; the BackupJob is only a thin HTTP client to it (seedocs/operations/backup-classes.md:51-67,:115). GC'ing the Secret when that BackupJob is deleted would break sidecar (re)mounts on the next Pod restart. - FoundationDB — the
backup_agentis a long-lived Deployment streaming continuously to S3 (docs/operations/backup-classes.md:49); the Secret must survive well past any single BackupJob.
Also note the GC semantics: ownerReferences for namespaced owners must be same-namespace, and the collector deletes a dependent only when all owners are gone — so any multi-owner approach must accumulate refs across active consumers, not overwrite.
Suggested direction
Tie the owner to the consumer's lifetime, not the backup's. Options to weigh:
- Owner = the application CR in the tenant namespace (Postgres/MariaDB/ClickHouse/Etcd/…), so the Secret lives as long as the workload that might need it and is GC'd when the app is deleted.
- For one-shot operator-side drivers (CNPG / MariaDB / Etcd), re-projection is idempotent, so transient GC between backups is harmless — these could safely owner-ref the backup if a tighter window is wanted.
- Leave ClickHouse
useSystemBucketand FoundationDB out of any backup-scoped GC entirely.
Decide per-driver (or scope GC to the one-shot drivers and document the exception) before implementing.
Acceptance
- Projected tenant Secrets get GC'd when their chosen owner is removed.
- ClickHouse
useSystemBucketand FoundationDB flows keep working across Pod restarts (covered by e2e).
Activity