Repository navigation
Tenant addon CRD HelmReleases remediate by uninstall, and their CRDs carry no resource-policy: keep #3555
Description
Activity
I went through CI artifacts looking for the occurrence this issue says is not shown. The sample is ten recent red e2e runs, nine nightly and one from a pull request. Seven had a usable instrument. One died during install, so no tenant clusters ever existed and there was nothing to observe. Two had no artifacts at all.
The remediation-by-uninstall path is not theoretical in this repo. It fires in all seven runs, on sibling releases:
Normal UninstallSucceeded helmrelease/kubernetes-test-previous-version-cilium Helm uninstall remediation for release cozy-cilium/cilium.v1 with chart [email protected]+02ff94ddaf23 succeeded. Each run has two or three distinct occurrences, all oncilium, and one run also has them oncsi. Counting needs care: the diagnostics block prints the event dump more than once per run, so the same occurrence shows up again under a different age, and a naive line count roughly doubles it.Those two releases are configured the same way as the ones in the table. In
packages/apps/kubernetes/templates/helmreleases/,cilium.yamlandcsi.yamlboth carryinstall.remediation.retries: -1andupgrade.remediation.retries: -1, same asgateway-api-crds.yaml,vertical-pod-autoscaler-crds.yamlandvolumesnapshot-crd.yaml. What separates them in these runs is how often the install fails. The configuration is the same.No uninstall event shows up on any release in the table, but for half of them that says nothing. The tenant addon set is identical in every run I looked at:
cilium,coredns,csi,ingress-nginx,metrics-server,ouroboros,prometheus-operator-crds,vsnap-crd. Sogateway-api-crdsandvertical-pod-autoscaler-crdsnever get installed in an e2e tenant at all, and CI cannot observe them either way.prometheus-operator-crdsandvsnap-crdare installed and visible to the instrument, both showInstallSucceededin the same dumps, and neither shows an uninstall.Three limits on what these artifacts can answer, all measured. Events are the only surviving trace: in a full bundle of 6337 files the string is in exactly one of them, the host event dump, and the oldest event there is 59 minutes old in a run that lasted over two hours.
HelmReleaseobjects are missing from the bundle completely. No file matcheskind: HelmRelease, and although the collector's ownapis.jsonlistshelm.toolkit.fluxcd.io, no namespace directory contains that group, so.status.historyis never collected. Snapshots are taken once per failed test, at failure time, so there is no second window where a custom resource could be seen to disappear.Correction to my previous comment. I claimed
HelmReleaseobjects are never collected into the report bundle, and that.status.historyis therefore unavailable. That is wrong. It holds for two of the six bundles I have, where that API group is missing from an otherwise complete collection, but the other four carry between 108 and 242HelmReleaseobjects each, including the tenant addon releases. I checked a single bundle and generalized from it.With the history available the check this issue actually wants can be run, and it comes out the same way the events did, only on firmer ground. Across all six bundles exactly four
HelmReleaseobjects carry an uninstall entry in.status.history, and all four arecilium:action: uninstall-remediation,status: uninstalling,deleted: 2026-08-04T05:22:26Z. No release from the table has one.kubernetes-test-latest-version-vsnap-crd, for example, carries a single entry,action: installwithstatus: deployed.The conclusion is unchanged and now rests on the durable record instead of events alone. What I wrote about event lifetime and about there being one snapshot window per failed test still holds.
- added a commit that references this issue
on Aug 7, 2026 The keep half of this landed: #3586 stamps resource-policy: keep on the four addon CRD packages, #3590 pins the annotation with tests, and #3595 makes the vendoring step fail loudly when the stamp misses instead of shipping unprotected CRDs. Remediation-by-uninstall is still there: csi got RetryOnFailure in #3581, the other addons still uninstall on a failed install.
- added a commit that references this issue
on Aug 8, 2026 - added and removed
on Sep 23, 2026 One more thing this chart needs is a test that every tenant addon release either retries in place or remediates without limit. The e2e HelmRelease history guard from #3594 depends on that, and nothing holds it today. The guard fails a run on a
failedSnapshot only when the release setsinstall.strategy.name: RetryOnFailure, and prints a note for every other addon. For an addon withremediation.retries: -1that is fine, since remediation is its declared recovery path.An addon added later with
retries: 0or no remediation block would slip through. That release never remediates, so itsfailedSnapshot is the only record of the failure. If the suite does not wait on it, the run stays green. The comment that states the assumption is athack/e2e-chainsaw/_lib/run-kubernetes.sh:5204on b42805b.The property holds on main at 94abf69.
cilium.yaml,csi.yamlandcsi-proxmox.yamlsetRetryOnFailure. Every file inpackages/apps/kubernetes/templates/helmreleases/setsretries: -1and no other retries value. The only tests that read these fields arecilium_install_retry_test.yamlandcsi_install_retry_test.yaml, and they cover those two releases only.The fix is a helm-unittest test in
packages/apps/kubernetes/tests/that renders every addon and requires each HelmRelease to carryRetryOnFailureorretries: -1on both actions. I'm adding this here because the fix this issue proposes changes the same fields on the CRD addons, and the test would hold whichever direction lands.- added a commit that references this issue
on Oct 5, 2026
Four tenant addon HelmReleases in
packages/apps/kubernetes/templates/helmreleases/ship CRDs from a chart'stemplates/directory and carryremediation.retries: -1with noinstall.strategy. Helm owns anything rendered fromtemplates/, so an uninstall deletes it, and deleting a CRD makes the API server garbage-collect every custom resource of that kind. The default install strategy remediates a failed install by uninstalling the release.Only a failed install gets there. A failed upgrade does not: with
Remediationunset,UpgradeRemediation.GetStrategy()returns the rollback strategy (api/v2/helmrelease_types.go:957-961), so a failed chart bump rolls back and leaves the CRDs. An absent release does not either.retries: -1never exhausts, so the absent branch returns an install instead of refusing one (internal/reconcile/atomic_release.go:389). The exposure is that this reinstall can fail, and then it is an install failure. Line references are helm-controller v1.5.0, the version this repo installs (internal/fluxinstall/manifests/fluxcd.yaml:8097).Second limiter. On a first install no custom resources exist, so a teardown destroys nothing. To lose anything the install has to run while custom resources already exist, which means the release went absent after a successful install (deleting its Helm storage Secret is the simplest way there) and the reinstall then failed. Whether that combination has ever occurred is not shown here. I collected no evidence of an occurrence, and the instrument that would record one is limited anyway: the e2e remediation-cycle guard watches only the parent HelmRelease and treats a failed attempt the same as a teardown (#3553).
resource-policy: keepdependsOntimeoutgateway-api-crdspackages/system/gateway-api-crds/templates/(1 file)vertical-pod-autoscaler-crdspackages/system/vertical-pod-autoscaler-crds/templates/(1 file)prometheus-operator-crdscrds, CRDs in itstemplates/(10 files)volumesnapshot-crdpackages/system/vsnap-crd/templates/(3 files)cert-manager-crdspackages/system/cert-manager-crds/templates/(6 files)crds.keep: trueby defaultcert-manager-crdsis in the table as the precedent. Every one of its CRD templates is annotated, so an uninstall leaves the definitions and their custom resources alone. That shape already exists in the tree.Two entries inherit the HelmRelease default timeout of 5m instead of setting 10m (
api/v2/helmrelease_types.go:110,GetTimeoutat:1503-1504), so they have the narrowest budget and no protection. None of the five declaresdependsOn. The dependency gate does protect the other tenant addons: a release waiting on a dependency returns before the release reconciler runs (internal/controller/helmrelease_controller.go:262) and never increments a failure count, sinceIncrementFailureCountis only reached from the install, upgrade and test reconcilers. The five withoutdependsOnare the ones that gate misses.corednsis a lighter case. Same remediation-only shape, and it is the tenant cluster's only DNS, so an uninstall drops the Deployment and its Service while every running pod keeps resolving against an address that stopped routing. The chart pins the Service address (packages/apps/kubernetes/templates/helmreleases/coredns.yaml,cozystack.defaultCoreDNSValues), so a reinstall comes back on the same address and pods recover without a restart. Without the pin a reallocated address would leave every existing pod pointing at a dead one for good. It also declaresdependsOnon the CNI, so the gate covers that failure path.Two directions, and they are not exclusive. Annotating the CRD templates with
helm.sh/resource-policy: keepfollowscert-manager-crdsand stops the deletion, at the cost of leaving definitions behind on a deliberate uninstall. Settinginstall.strategy.name: RetryOnFailureon both actions follows #3552 and takes the teardown out of the retry path without changing what a deliberate uninstall does. #3554 covers a related trap on the remediation values that the second direction has to account for.