You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
tenant: monitoring and gateway releases lose telemetry and the ACME account to install remediation #3624
Two more tenant child HelmReleases remediate a failed install by uninstall, and their post-delete hooks destroy state that a merely slow install never threatened. This is the same shape as the harbor fix (#3621) and the etcd/seaweedfs pair that followed it: install.remediation.retries: -1 with no strategy falls back to RemediateOnFailure, remediation for an install is an uninstall in every form the API offers, spec.uninstall.disableHooks is unset (default false), so the post-delete hooks run against a release that was healthy apart from missing a readiness budget.
The two releases
packages/apps/tenant/templates/monitoring.yaml carries no remediation strategy, and packages/extra/monitoring/templates/hooks/cleanup.yaml:57 deletes PVCs by the apps.cozystack.io/application.name=<release>-system label. A budget miss on a slow first install therefore deletes the tenant's metrics and log volumes. Telemetry is not control-plane state, but it is unrecoverable.
packages/apps/tenant/templates/gateway.yaml carries no strategy either, and its cleanup hook deletes the cozystack-acme-account secret, the ACME account key. The key regenerates on the next install, but re-registration runs into the Let's Encrypt rate limits, so repeated remediation cycles can lock certificate issuance out for the tenant for hours.
Fix shape
Same as #3621 and the etcd/seaweedfs pair: RetryOnFailure on install and upgrade remediation for both releases, pinned by an install_retry test in packages/apps/tenant/tests/. The hooks themselves are correct for an honest uninstall and should not change; the defect is remediation invoking them for a release that only needed more time.
For the wider inventory: roughly two dozen more nested HelmReleases across the catalog still carry retries: -1 on the default strategy without owning volumes. Those are a lower-stakes decision (a retry loop versus a clean slate, no data at risk) and deliberately not part of this issue.
Activity
fix(tenant): retry failed monitoring and gateway installs in place (#…
lexfrei commentedon Sep 23, 2026
Fixed by #3633: the tenant monitoring and gateway releases now use
RetryOnFailurefor install and upgrade.