The status.history guard in hack/e2e-chainsaw/_lib/run-kubernetes.sh (cozy_guard_helmrelease, via helmrelease_has_remediation_cycle in hack/e2e-chainsaw/_lib/remediation-guard.sh) fails a run on a failed or uninstalled Snapshot of a RetryOnFailure release. It misses the case it is most often cited for: an install that fails once, for example on the install timeout, and recovers on the next attempt.
Mechanism, at helm-controller v1.5.0 (the version internal/fluxinstall/manifests/fluxcd.yaml installs) and Helm v4.1.1 (the version it builds against):
- Under
RetryOnFailure, a failed release is retried as an upgrade (internal/reconcile/atomic_release.go:469-474).
- With no deployed release, Helm upgrades from the last release, which is the failed install (
pkg/action/upgrade.go:258-262), and marks it superseded once the upgrade succeeds (:511-512).
- helm-controller records both observations and replaces the old Snapshot in place (
internal/reconcile/release.go:96-110).
The history then reads deployed, superseded, which by status looks the same as an ordinary upgrade, and the guard passes. A failed Snapshot survives only when a deployed release already existed (a failed upgrade), or when the install failed at least twice in a row.
Two places still claim otherwise and predate the guard rework:
docs/agents/e2e-testing.md:77 says a timed-out attempt leaves a failed Snapshot that the guard fails on, "so a success arriving from a later attempt is one the suite has already failed on". After a single timed-out install that the next attempt recovers, the Snapshot is superseded and the suite passes.- The predicate itself, which matches statuses only.
Detection needs a signal beyond the status list. Counting history entries does not work: the OIDC path of run_kubernetes_test upgrades the parent on purpose (cozy_switch_and_assert_oidc_custom_config) before the guard runs. Candidates to evaluate against a real run: a superseded Snapshot whose successor has the same chartVersion, configDigest and ociDigest (a retry changes neither chart nor values, while a spec-driven upgrade changes the config digest), or the InstallFailed event on the HelmRelease, which is lossy and ages out. Either needs a measured run before it can fail a run, so that the guard does not go red on a legitimate upgrade.
hack/run-kubernetes-remediation_test.bats has a test, "single-release guard cannot see a failed install the next attempt recovered", that pins the current pass. It is the test to flip when detection lands.
The
status.historyguard inhack/e2e-chainsaw/_lib/run-kubernetes.sh(cozy_guard_helmrelease, viahelmrelease_has_remediation_cycleinhack/e2e-chainsaw/_lib/remediation-guard.sh) fails a run on afailedoruninstalledSnapshot of aRetryOnFailurerelease. It misses the case it is most often cited for: an install that fails once, for example on the install timeout, and recovers on the next attempt.Mechanism, at helm-controller v1.5.0 (the version
internal/fluxinstall/manifests/fluxcd.yamlinstalls) and Helm v4.1.1 (the version it builds against):RetryOnFailure, a failed release is retried as an upgrade (internal/reconcile/atomic_release.go:469-474).pkg/action/upgrade.go:258-262), and marks itsupersededonce the upgrade succeeds (:511-512).internal/reconcile/release.go:96-110).The history then reads
deployed,superseded, which by status looks the same as an ordinary upgrade, and the guard passes. AfailedSnapshot survives only when a deployed release already existed (a failed upgrade), or when the install failed at least twice in a row.Two places still claim otherwise and predate the guard rework:
docs/agents/e2e-testing.md:77says a timed-out attempt leaves afailedSnapshot that the guard fails on, "so a success arriving from a later attempt is one the suite has already failed on". After a single timed-out install that the next attempt recovers, the Snapshot issupersededand the suite passes.Detection needs a signal beyond the status list. Counting history entries does not work: the OIDC path of
run_kubernetes_testupgrades the parent on purpose (cozy_switch_and_assert_oidc_custom_config) before the guard runs. Candidates to evaluate against a real run: asupersededSnapshot whose successor has the samechartVersion,configDigestandociDigest(a retry changes neither chart nor values, while a spec-driven upgrade changes the config digest), or theInstallFailedevent on the HelmRelease, which is lossy and ages out. Either needs a measured run before it can fail a run, so that the guard does not go red on a legitimate upgrade.hack/run-kubernetes-remediation_test.batshas a test, "single-release guard cannot see a failed install the next attempt recovered", that pins the current pass. It is the test to flip when detection lands.