Skip to content

tenant: monitoring and gateway releases lose telemetry and the ACME account to install remediation #3624

Closed

Description

Summary

Two more tenant child HelmReleases remediate a failed install by uninstall, and their post-delete hooks destroy state that a merely slow install never threatened. This is the same shape as the harbor fix (#3621) and the etcd/seaweedfs pair that followed it: install.remediation.retries: -1 with no strategy falls back to RemediateOnFailure, remediation for an install is an uninstall in every form the API offers, spec.uninstall.disableHooks is unset (default false), so the post-delete hooks run against a release that was healthy apart from missing a readiness budget.

The two releases

packages/apps/tenant/templates/monitoring.yaml carries no remediation strategy, and packages/extra/monitoring/templates/hooks/cleanup.yaml:57 deletes PVCs by the apps.cozystack.io/application.name=<release>-system label. A budget miss on a slow first install therefore deletes the tenant's metrics and log volumes. Telemetry is not control-plane state, but it is unrecoverable.

packages/apps/tenant/templates/gateway.yaml carries no strategy either, and its cleanup hook deletes the cozystack-acme-account secret, the ACME account key. The key regenerates on the next install, but re-registration runs into the Let's Encrypt rate limits, so repeated remediation cycles can lock certificate issuance out for the tenant for hours.

Fix shape

Same as #3621 and the etcd/seaweedfs pair: RetryOnFailure on install and upgrade remediation for both releases, pinned by an install_retry test in packages/apps/tenant/tests/. The hooks themselves are correct for an honest uninstall and should not change; the defect is remediation invoking them for a release that only needed more time.

For the wider inventory: roughly two dozen more nested HelmReleases across the catalog still carry retries: -1 on the default strategy without owning volumes. Those are a lower-stakes decision (a retry loop versus a clean slate, no data at risk) and deliberately not part of this issue.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/tenantIssues or PRs related to the tenant chart and multi-tenancy
    on Aug 7, 2026
  2. lexfrei commented on Sep 23, 2026

    @lexfrei
    ContributorAuthor

    Fixed by #3633: the tenant monitoring and gateway releases now use RetryOnFailure for install and upgrade.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/tenantIssues or PRs related to the tenant chart and multi-tenancykind/bugCategorizes issue or PR as related to a bugtriage/not-reproducibleIndicates an issue can not be reproduced as described

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions