Skip to content

e2e: no remediation-cycle guard covers the tenant addon HelmReleases, and the existing helper cannot tell a retry from a teardown #3553

Closed

Description

helmrelease_has_remediation_cycle in hack/e2e-chainsaw/_lib/remediation-guard.sh catches a HelmRelease that fell into an install or upgrade remediation cycle. It reads .status.history and flags a failed or uninstalled Snapshot, deliberately rather than .status.installFailures / .status.upgradeFailures, since ClearFailures() zeroes those on every successful reconcile and a guard reading them after the release is Ready is vacuous.

The helper is wired, and it is wired exactly as documented. hack/e2e-chainsaw/_lib/run-kubernetes.sh sources it at line 3 and calls it at line 1039 against the parent kubernetes-${test_name} HelmRelease, and both places the documentation mentions it scope it to that same object: docs/agents/e2e-testing.md:60 heads the section "Assert the parent HelmRelease did not remediate", and :125 asks for the guard when a change "touches parent-HR behavior". So this is not an instrument that was left unconnected or pointed the wrong way.

The gap is that nothing covers the tenant addon HelmReleases the parent installs, and no guidance asks for it. A remediation cycle on one of those leaves no mark that the suite inspects, so a run surfaces only the downstream symptom, such as a node-readiness deadline, with nothing identifying the teardown that preceded it. The CNI release is the sharpest case, because it is the one whose teardown removes a cluster-wide precondition rather than churning a single component, but the same blindness applies to every addon.

Extending the coverage needs more than another release name, because the helper's signal is ambiguous under the newer action strategy. Treating failed and uninstalled as one signal is correct for a release under the default RemediateOnFailure, where a failed attempt is followed by a teardown. It is wrong for a release under RetryOnFailure, where a failed attempt leaves the applied manifests in place and is retried as an upgrade: there a failed Snapshot is the normal non-destructive path and only uninstalled indicates a teardown. Pointed at such a release as it stands, the guard would report a remediation cycle on the first failed attempt even though nothing was removed.

The two parts are separable and the second gates the first: the helper has to distinguish a teardown from a plain failure before its coverage can reach releases that no longer remediate by uninstalling.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/testingIssues or PRs related to testing (e2e, bats, unit tests)
    triage/acceptedIndicates an issue is ready to be actively worked on
    priority/backlogGeneral backlog priority. Lower than priority/important-longterm
    and removed
    triage/needs-triageIndicates an issue needs triage by a maintainer
    on Sep 23, 2026
  2. lexfrei commented on Sep 28, 2026

    @lexfrei
    ContributorAuthor

    The second part of this issue doesn't hold, so #3594 does not split the reading. The issue assumes that under RetryOnFailure a failed Snapshot is harmless and only uninstalled means a teardown. In helm-controller v1.5.0 a release with an active retry strategy never remediates: internal/reconcile/atomic_release.go returns an upgrade for a failed release before it reaches the remediation switch. The other two uninstall paths, a changed release target and deletion, clear history afterwards. So uninstalled can't appear on such a release, and failed is the only footprint a failed action leaves.

    What #3594 does instead is extend the guard to the tenant addon releases. A failed or uninstalled Snapshot fails the run on the parent, cilium and csi, which retry in place, and is printed as a note on the default-strategy addons. The same classification also runs on the failure paths that exit before the guard.

  3. added a commit that references this issue on Oct 5, 2026
    415430b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/bugCategorizes issue or PR as related to a bugpriority/backlogGeneral backlog priority. Lower than priority/important-longtermtriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions