Skip to content

Garbage-collect projected cozy-backups-creds Secrets via ownerReferences #2734

Open

Description

Background

Follow-up to #2716.

The credentials projector (internal/backupcontroller/credentials_projector.go) copies the platform-managed S3 Secret into a tenant namespace as cozy-backups-creds, on demand right before each BackupJob/RestoreJob runs (backupjob_controller.go:150, restorejob_controller.go:136). The projected Secret carries the apps.cozystack.io/managed-by label but no ownerReferences. As a result, once a tenant runs its last backup the projected Secret lingers in the tenant namespace indefinitely, holding usable platform S3 credentials with no lifecycle tied to anything.

Proposal

Set ownerReferences on the projected Secret so the Kubernetes garbage collector reclaims it once its owner is gone, shrinking the window in which platform credentials sit idle in tenant namespaces. This is complementary to — not a replacement for — the existing managed-by ownership guard (credentials_projector.go:205-213): the label prevents clobbering a Secret the projector doesn't own; ownerReferences drive cleanup.

Design constraint — do NOT owner-ref the triggering BackupJob

The obvious "GC when the backup is done" (owner = the BackupJob) is unsafe, because the single cozy-backups-creds Secret is shared by consumers that outlive a one-shot backup:

  • ClickHouse useSystemBucket — the clickhouse-backup sidecar runs inside the long-lived ClickHouse Pod and mounts cozy-backups-creds; the BackupJob is only a thin HTTP client to it (see docs/operations/backup-classes.md:51-67, :115). GC'ing the Secret when that BackupJob is deleted would break sidecar (re)mounts on the next Pod restart.
  • FoundationDB — the backup_agent is a long-lived Deployment streaming continuously to S3 (docs/operations/backup-classes.md:49); the Secret must survive well past any single BackupJob.

Also note the GC semantics: ownerReferences for namespaced owners must be same-namespace, and the collector deletes a dependent only when all owners are gone — so any multi-owner approach must accumulate refs across active consumers, not overwrite.

Suggested direction

Tie the owner to the consumer's lifetime, not the backup's. Options to weigh:

  • Owner = the application CR in the tenant namespace (Postgres/MariaDB/ClickHouse/Etcd/…), so the Secret lives as long as the workload that might need it and is GC'd when the app is deleted.
  • For one-shot operator-side drivers (CNPG / MariaDB / Etcd), re-projection is idempotent, so transient GC between backups is harmless — these could safely owner-ref the backup if a tighter window is wanted.
  • Leave ClickHouse useSystemBucket and FoundationDB out of any backup-scoped GC entirely.

Decide per-driver (or scope GC to the one-shot drivers and document the exception) before implementing.

Acceptance

  • Projected tenant Secrets get GC'd when their chosen owner is removed.
  • ClickHouse useSystemBucket and FoundationDB flows keep working across Pod restarts (covered by e2e).

Activity

  1. added
    kind/cleanupCategorizes issue or PR as related to cleanup of code, process, or technical debt
    area/storageIssues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor)
    on May 26, 2026
  2. added
    triage/acceptedIndicates an issue is ready to be actively worked on
    priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete
    on Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area/storageIssues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor)kind/cleanupCategorizes issue or PR as related to cleanup of code, process, or technical debtpriority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to completetriage/acceptedIndicates an issue is ready to be actively worked on

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions