Repository navigation
etcd (tenant module): HelmRelease stuck Terminating + orphaned PVCs/TLS secrets after the module is disabled #3090
Copy link
Copy link
Closed
Labels
area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)kind/bugCategorizes issue or PR as related to a bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releaseMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked onIndicates an issue is ready to be actively worked on
Description
Describe the bug
After enabling and then disabling the etcd tenant module, the etcd HelmRelease gets stuck in Terminating (the Flux finalizer is never removed), and its data PVCs and TLS secrets are left orphaned in the tenant namespace.
Environment
- Cozystack: dev9 (config version 45, charts @
fe13212a10a6) - Provider: on-prem (bare-metal)
To Reproduce
- Create a tenant with
etcd: true; wait until Ready. - Set
etcd: false; wait. kubectl -n <tenant-ns> get helmrelease etcd— stuck withdeletionTimestampandfinalizers: ["finalizers.fluxcd.io"](observed 12+ minutes).kubectl -n <tenant-ns> get pvc,secret | grep etcd— PVCs and TLS secrets remain.
Expected behaviour
Disabling the module removes the HelmRelease cleanly and leaves no orphaned PVCs/secrets.
Actual behaviour
helmrelease/etcd (Terminating, finalizers.fluxcd.io, 12m+)
pvc/data-etcd-0 Bound
pvc/data-etcd-1 Bound
pvc/data-etcd-2 Bound
secret/etcd-ca-tls
secret/etcd-client-tls
secret/etcd-peer-ca-tls
secret/etcd-peer-tls
secret/etcd-server-tls
Additional context
- Root cause (stuck HR): When
etcd: false, the tenant chart prunes theetcdHelmRelease. The workload (StatefulSet/pods) is removed, but the HelmRelease CR hangs: after thedeletionTimestampis set, the (sharded) helm-controller stops reconciling this HR and never removesfinalizers.fluxcd.io. The HR is sourced viachartRef(cozystack-etcd-application-default-etcd) — the chart source is likely pruned together with the HR, so helm-controller can no longer load the chart to complete the uninstall. (Same class as the "stuck-HR finalizer" platform observation.) - Root cause (PVCs): the etcd StatefulSet
volumeClaimTemplates(data-etcd-*) are never garbage-collected on StatefulSet deletion, and nopersistentVolumeClaimRetentionPolicy/cleanup exists. - Root cause (TLS):
etcd-*-tlssecrets are not owner-referenced, so they orphan. - Note: A full
kubectl delete tenanttears down cleanly (the tenantcleanup-jobforce-deletes HelmReleases and the namespace+PVs are removed). The leak is specific to the per-module toggle-off path, where the namespace persists. - Proposed fix: (1) investigate why the Flux finalizer isn't removed when a
chartRef-based HR is pruned (source-vs-HR deletion ordering); (2) addpersistentVolumeClaimRetentionPolicy/PVC cleanup for the etcd data PVCs; (3) clean up the orphaned TLS secrets.
Found during a create→delete leak sweep of packages/extra tenant modules on Cozystack dev9 (toggle module on → off; the tenant namespace persists). Companion to the apps sweep (#3056–#3062).
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)kind/bugCategorizes issue or PR as related to a bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releaseMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked onIndicates an issue is ready to be actively worked on
Activity
fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant modu…
fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant modu…