Repository navigation
Tenant csi HelmRelease remediates by uninstall, and its 10m install budget is spent pulling latest-tagged images #3576
Description
packages/apps/kubernetes/templates/helmreleases/csi.yaml carries remediation.retries: -1 on install and upgrade (:24-30) with no install.strategy, so a failed install is remediated by uninstalling the release. That is the same configuration #3552 changes for cilium. The general form is known and covers every tenant addon HelmRelease the chart renders; this issue is about the one chain that has now been seen ending a suite on its own.
What was observed
Run 31025557528 on 4d16f4fdb, job E2E Tests, suite kubernetes-previous, failing at hack/e2e-chainsaw/_lib/run-kubernetes.sh:677:
error: timed out waiting for the condition on helmreleases/kubernetes-test-previous-version-csi
That run carries #3552, so the tenant cilium release no longer remediates by uninstall. Its events are InstallFailed and then UpgradeSucceeded with nothing torn down between them, both tenant workers reach Ready, and node join passes for the first time on that branch. With cilium no longer ending the suite earlier, csi is what ends it.
The csi events from the same run, ages as printed in a dump taken at 18:52:36:
6m20s Warning InstallFailed helmrelease/kubernetes-test-previous-version-csi Helm install failed ... timeout waiting for: [DaemonSet/cozy-csi/kubevirt-csi-node status: 'InProgress']
6m20s Normal UninstallSucceeded helmrelease/kubernetes-test-previous-version-csi Helm uninstall remediation for release ...
Three things compound
Times below are the dump timestamp minus the printed event age, and carry that granularity.
dependsOn starts the clock too early. csi.yaml:60 depends on <release>-cilium, so the install begins when cilium reports Ready, at roughly 18:36:36. The tenant workers reached Ready at roughly 18:44, from node ages of 4m41s and 5m30s read at 18:49. kubevirt-csi-node is a DaemonSet, so for about eight of its ten minutes there were no nodes to schedule on and no way for the rollout to progress.
The strategy turns a slow install into a restart. With no install.strategy.name: RetryOnFailure next to the retries: -1 that never exhausts (csi.yaml:24-27), the install that failed at 18:46:16 is remediated by an uninstall, and the next attempt starts from a torn-down release instead of resuming a rollout that had finally acquired nodes.
The suite budget cannot absorb one retry. csi.yaml:12 sets timeout: 10m, and run-kubernetes.sh:677 waits --timeout=10m on the same release. The two are equal, so the first attempt to exhaust its own timeout is terminal for the suite whatever the retry does next. The uninstall at 18:46:16 falls inside the suite window, which ran from 18:39:29 to 18:49:29.
Relation to #3555
#3555 records that the dependency gate protects the tenant addons declaring dependsOn, since a release waiting on a dependency returns before the release reconciler runs and never increments a failure count. That holds, and this narrows it: the gate protects only while the dependency is not ready. csi fails right after the gate opens, and it fails partly because the gate opened, since that is the moment its own timeout starts counting while the workers still do not exist. A release whose readiness depends on nodes needs a budget that starts from something later than the CNI.
Shape of a fix
#3552 is the precedent for the strategy part, on both actions for the reason given there. The other two are independent of it. The budget equality is a test-side choice, and the early clock is a property of what csi waits for rather than of the remediation strategy. Fixing only the strategy leaves a csi install that still spends most of its timeout before a node exists.
Activity
lexfrei commentedon Aug 6, 2026
Two claims in the body do not survive a recheck. Both are corrected below, and the recheck found the actual cause.
The first is "for about eight of its ten minutes there were no nodes to schedule on". The tenant snapshot has the DaemonSet creating both pods at 18:36:16 and both
Scheduledin the same second, ontomd0-ncqcf-7zdx6andmd0-ncqcf-txw2m. Nodes were present for the whole window. That claim came from reading a nodeAGEcolumn against the timestamp of the chainsaw block containing it. Chainsaw buffers a script's stdout and emits it once when the script exits, so every line in the block carries the exit time and a relative age read against it means nothing.run-kubernetes.sh:539prints that table right after the node-join gate at:462, minutes before the csi wait at:677.The second is "the suite budget cannot absorb one retry". The csi events are
InstallFailedandUninstallSucceededat 18:46:16, thenInstallSucceededat 18:46:43. Teardown and reinstall cost 27 seconds and finished 2m46s before the suite wait expired at 18:49:29.What consumed the ten minutes is image pulls. Both pods were scheduled at 18:36:16 and then sat fetching. On one node
quay.io/openshift/origin-csi-node-driver-registrar:latesttook 2m16.833s, on the other 4m1.224s, and thekubevirt-csi-driverimage ahead of it took about five minutes there. The second pod started its last container at 18:46:22, six seconds after the install timeout fired at 18:46:16, and the uninstall killed it at 18:46:23. Both registrar and livenessprobe arelatesttags from a public registry, atpackages/system/kubevirt-csi-node/templates/deploy.yaml:204and:228and again atpackages/apps/kubernetes/templates/csi/deploy.yaml:141, so every tenant install fetches them fresh.The release still never went Ready, and that part is not explained by the evidence. The reinstall created new pods at 18:46:28; both stayed
Pendingwith all three containers inContainerCreatingandPodReadyToStartContainers: Falsethrough to 18:49:29, emitting no events in those three minutes, and the DaemonSet readdesiredNumberScheduled: 2, numberReady: 0, numberUnavailable: 2. The images were already local by then, so this is not a pull. The tenant crust-gather's 180s window closed at 18:49:29 and the tenant was torn down, so what held the sandbox is not recoverable from this run.Two things stand. The remediation shape is real, and here it tore down a rollout that was six seconds from complete. And the timers are offset: the install began at 18:36:16 when the dependency gate opened while the suite's wait began at 18:39:29, so with both budgets at 10m the suite only ever sees the install's last 6m47s.
[-]Tenant csi HelmRelease remediates by uninstall, and its 10m budget starts at CNI-ready, before any worker registers[/-][+]Tenant csi HelmRelease remediates by uninstall, and its 10m install budget is spent pulling latest-tagged images[/+]fix(kubernetes): retry the tenant CSI install instead of uninstalling…
fix(tenant): retry failed etcd and seaweedfs installs in place (#3630)
lexfrei commentedon Sep 2, 2026
Status check against main. The remediation half is fixed: #3581 put RetryOnFailure on both install and upgrade in packages/apps/kubernetes/templates/helmreleases/csi.yaml. The two causes from the correction comment are still there: :latest pulls in packages/system/kubevirt-csi-node/templates/deploy.yaml and packages/apps/kubernetes/templates/csi/deploy.yaml (tracked in #3578), and the 10m install timeout in csi.yaml sits on the same 10m the e2e wait uses in hack/e2e-chainsaw/_lib/run-kubernetes.sh, with the release still gated on cilium. Keeping this open for those.