Skip to content

etcd (tenant module): HelmRelease stuck Terminating + orphaned PVCs/TLS secrets after the module is disabled #3090

Closed

Description

@scooby87

Describe the bug

After enabling and then disabling the etcd tenant module, the etcd HelmRelease gets stuck in Terminating (the Flux finalizer is never removed), and its data PVCs and TLS secrets are left orphaned in the tenant namespace.

Environment

  • Cozystack: dev9 (config version 45, charts @ fe13212a10a6)
  • Provider: on-prem (bare-metal)

To Reproduce

  1. Create a tenant with etcd: true; wait until Ready.
  2. Set etcd: false; wait.
  3. kubectl -n <tenant-ns> get helmrelease etcd — stuck with deletionTimestamp and finalizers: ["finalizers.fluxcd.io"] (observed 12+ minutes).
  4. kubectl -n <tenant-ns> get pvc,secret | grep etcd — PVCs and TLS secrets remain.

Expected behaviour
Disabling the module removes the HelmRelease cleanly and leaves no orphaned PVCs/secrets.

Actual behaviour

helmrelease/etcd                 (Terminating, finalizers.fluxcd.io, 12m+)
pvc/data-etcd-0   Bound
pvc/data-etcd-1   Bound
pvc/data-etcd-2   Bound
secret/etcd-ca-tls
secret/etcd-client-tls
secret/etcd-peer-ca-tls
secret/etcd-peer-tls
secret/etcd-server-tls

Additional context

  • Root cause (stuck HR): When etcd: false, the tenant chart prunes the etcd HelmRelease. The workload (StatefulSet/pods) is removed, but the HelmRelease CR hangs: after the deletionTimestamp is set, the (sharded) helm-controller stops reconciling this HR and never removes finalizers.fluxcd.io. The HR is sourced via chartRef (cozystack-etcd-application-default-etcd) — the chart source is likely pruned together with the HR, so helm-controller can no longer load the chart to complete the uninstall. (Same class as the "stuck-HR finalizer" platform observation.)
  • Root cause (PVCs): the etcd StatefulSet volumeClaimTemplates (data-etcd-*) are never garbage-collected on StatefulSet deletion, and no persistentVolumeClaimRetentionPolicy/cleanup exists.
  • Root cause (TLS): etcd-*-tls secrets are not owner-referenced, so they orphan.
  • Note: A full kubectl delete tenant tears down cleanly (the tenant cleanup-job force-deletes HelmReleases and the namespace+PVs are removed). The leak is specific to the per-module toggle-off path, where the namespace persists.
  • Proposed fix: (1) investigate why the Flux finalizer isn't removed when a chartRef-based HR is pruned (source-vs-HR deletion ordering); (2) add persistentVolumeClaimRetentionPolicy/PVC cleanup for the etcd data PVCs; (3) clean up the orphaned TLS secrets.

Found during a create→delete leak sweep of packages/extra tenant modules on Cozystack dev9 (toggle module on → off; the tenant namespace persists). Companion to the apps sweep (#3056–#3062).

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release
    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)
    on Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)kind/bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions