E2E kubernetes-previous fails consistently: Cilium install times out inside tenant cluster on v1.34.9 nodes #3229

Activity

1. IvanHunters commented on Jul 7, 2026

   

   IvanHunters

   on Jul 7, 2026

   CollaboratorAuthor

   More actions

   Update from CI artifact analysis (cozyreport from run 28843794670):

   cilium-agent container never wrote any log line (current.log / previous.log are both zero bytes). Pod state on the tenant node:

   initContainerStatuses:
   - name: config
     state: {waiting: {reason: PodInitializing}}
     imageID: ""                               # <-- pull hasn't produced an image ID
   - name: mount-cgroup
     state: {waiting: {reason: PodInitializing}}
     imageID: ""
   - name: apply-sysctl-overwrites
     state: {waiting: {reason: PodInitializing}}
     imageID: ""
   ... (all init containers stuck the same way)

   imageID: "" on every init container means kubelet is still pulling iad.ocir.io/idyksih5sir9/cozystack/cilium:main@sha256:16e9a5cd3b3614571329a309fc283f5a43e7522bef91d1ff539545c0ec1bc857 — nothing has landed on disk yet when the HR's 10-minute install timeout fires.

   Neither of my two candidates (Cilium 1.19.5 vs k8s 1.34 API, kubelet image pull latency) is quite right — the delta is not Cilium's startup, and it's not the kubelet binary's image. It's the Cilium container image itself taking >10 min to pull to a fresh Talos worker VM on this CI runner (iad.ocir.io mirror + fresh node = no image cache).

   latest (v1.35) took 13:10 — passed inside the 12 min until nodes Ready wall-clock the bats has, so the tolerance was ~30s. previous (v1.34) took 13:52 — 82 seconds over budget. This is a timing race directly against the HR timeout: 10m, not a version-specific bug. Both minors are equally likely to flake on the pull-window edge — v1.35 just got lucky first.

   Minimal fix: bump spec.timeout on the tenant Cilium HR from 10m to 15m in packages/apps/kubernetes/templates/helmreleases/cilium.yaml. Aligns with the outer HR budget the monitoring chart already uses (15m timeout — see #3176 discussion) and gives image pull enough headroom for slow CI-runner network conditions.

   Opening a PR against main with that one-line change.
2. IvanHunters commented on Jul 27, 2026

   

   IvanHunters

   on Jul 27, 2026

   CollaboratorAuthor

   More actions

   Update: the cilium install timeout is no longer previous-only. On the last three consecutive main nightlies it fails on latest (v1.35.6) as well, deterministically, on both cases each night:

   • run 30141652222 (2026-07-25): kubernetes-latest + kubernetes-previous
   • run 30185666660 (2026-07-26): kubernetes-latest + kubernetes-previous
   • run 30233887192 (2026-07-27): kubernetes-latest + kubernetes-previous

   Same signature, now on latest:

   tenant-test HelmRelease/kubernetes-test-latest-version-cilium InstallFailed:
   Helm install failed for release cozy-cilium/cilium ...
   timeout waiting for: [Deployment/cozy-cilium/cilium-operator status: 'InProgress']
   

   with coredns / csi / ingress-nginx / metrics-server all DependencyNotReady on cilium. The worker VM boots (VMI Running, guest AgentConnected=True) and the Kamaji control plane is Ready with the LB reachable, but the tenant worker Node never registers because the tenant cilium never reaches Ready.

   Because latest (v1.35.6) now reproduces the same failure this issue attributed to the v1.34.9 kubelet, the version-specific hypothesis (candidate root cause #1) is weakened: it is not unique to the previous minor. This now looks like a deterministic regression on main affecting both minors, pointing more at candidate #2 (cilium-operator never reaching Ready within the 10m Helm wait) or a newer common regression. Next step: bisect the main nightlies to localize when latest went red, and capture cilium-operator/agent logs from a repro to disambiguate.
3. lexfrei commented on Jul 28, 2026

   

   Aleksei Sviridkin (lexfrei)

   on Jul 28, 2026

   Contributor

   More actions

   #3231 is fixed and this still reproduces: 12 of 19 run-attempts I audited in the 07-25..07-28 window fail node-join, on both minors. Two things that are not in the thread yet.

   The HR waits on Deployment/cozy-cilium/cilium-operator, and the chart runs the operator with replicas: 2 plus a required podAntiAffinity on kubernetes.io/hostname. Available needs two schedulable nodes, not just a finished image pull on the first one. In run 30379656463 the first worker CSR was signed at 18:48:19 and the second at 18:50:56, which is 85s after the install had already timed out at 18:49:31.

   Flux also remediates a failed install by uninstalling the release: "Remediation, using an uninstall, is performed between each attempt", and .spec.install.strategy defaults to RemediateOnFailure. With retries: -1 every miss deletes the release and starts the rollout from the beginning, so the tenant cluster has no CNI at all for part of each cycle. In that run the uninstall completed at 18:49:37 and the next install started at 18:51:24, ten seconds before the node-join wait expired.

   I am not reopening the timeout question from #3230. spec.install.strategy.name: RetryOnFailure is a different knob: a failed install retries as an upgrade instead of uninstalling, so whatever converged so far is kept. The v2 CRD we run already accepts that field.
4. lexfrei commented on Jul 29, 2026

   

   Aleksei Sviridkin (lexfrei)

   on Jul 29, 2026

   Contributor

   More actions

   Reproduced on a clean base overnight: nightly run 30418650865 on e6460dbef, no PR patch involved. Both heavy suites fail node-join, the marker reads within 12m, and the tenant cilium HelmRelease logs a failed install on both clusters.

   Timings have the same shape as the run in my comment above. Both apiserver pods run a talos-csr-signer sidecar, so the deltas below are measured from the first one that started listening:

   • kubernetes-previous: signers up at 04:38:29 and 04:38:32, one CSR at 04:49:07, +10m38s. The second worker did not request a certificate inside the window.
   • kubernetes-latest: signers up at 05:11:11 and 05:11:13, CSRs at 05:21:41 and 05:22:24, +10m30s and +11m13s.

   Both first-CSR delays sit in the upper half of the 8m53s to 11m46s spread I quoted, against a 10m install budget. When the diagnostics were collected, both cilium HelmReleases were 13-14m old and back in Running 'install' action with timeout of 10m0s, so already on a retry after the first attempt had been remediated.

   Nothing else failed in that run.
5. lexfrei commented on Jul 29, 2026

   

   Aleksei Sviridkin (lexfrei)

   on Jul 29, 2026

   Contributor

   More actions

   Correction to my earlier comment in this thread.

   I checked the live objects instead of the chart. On the tenant side the cilium-operator Deployment runs with replicas: 1, and the scheduler message on its pod is no nodes available to schedule pods. That message is the stronger evidence of the two: it says zero nodes, not one. With two replicas and one node the scheduler would have complained about anti-affinity instead.

   So the part of my comment that explained the failure through replicas: 2 plus a required podAntiAffinity is wrong and I withdraw it, along with the half of the recommendation that suggested setting operator.replicas: 1. The umbrella values already set it to 1. The 2 I quoted comes from the vendored upstream chart, which is the input to the render rather than the effective configuration.

   What does not change: the timeline, the conclusion that the install was already late against its own 10 minute budget, and the remediation that uninstalls the release and so takes the CNI away from a node that had it.

   What gets sharper: what binds is the existence of the first tenant node, not the readiness of a second one. On the run I timed, the first certificate was signed 10m27s after the HelmRelease was created, so the install was 27 seconds past its own budget before any node existed at all.

   That leaves one change worth making, and it is not about replicas. Setting spec.install.strategy.name: RetryOnFailure on the tenant cilium HelmRelease makes an expired install retry instead of uninstalling, so a node that already has the CNI keeps it. It is worth setting on upgrade as well, matching what the operator already does for the releases it generates: internal/operator/package_reconciler.go sets RetryOnFailure on both install and upgrade. The tenant releases rendered from the kubernetes chart never got that policy.
6. lexfrei commented on Aug 5, 2026

   

   Aleksei Sviridkin (lexfrei)

   on Aug 5, 2026

   Contributor

   More actions

   #3552 switches the tenant Cilium HelmRelease to RetryOnFailure on both the install and the upgrade action, so a failed install is retried in place rather than uninstalled between attempts. For this issue that removes one amplifier and nothing else: once the 10m install budget is exceeded, the default remediation uninstalls the release, which takes the CNI away for the window until the next install applies the manifests again. It does not touch why the install exceeds its budget, so it should not be expected to make this test pass. One practical consequence for reading future runs: with the strategy in place there is no UninstallSucceeded event any more, and the release stays failed and retries every 30s instead.
7. lexfrei commented on Aug 7, 2026

   

   Aleksei Sviridkin (lexfrei)

   on Aug 7, 2026

   Contributor

   More actions

   Six more runs from last night, three PRs times two E2E attempts, all on current main. kubernetes-previous failed 5 of 6, same signature as above: tenant cilium HR fails with "failed early due to stalled resources", no CNI, node-join ends below 2 nodes in 18m. kubernetes-latest failed 6 of 6, and one retry showed a second leg: cilium came up, nodes joined, the test workload served traffic, then the suite timed out on "waiting for the condition on helmreleases/-csi". So #3552 covers the cilium leg but not the csi one. Engine suites in the same runs passed every time, this is not branch-specific.
8. lexfrei commented on Aug 7, 2026

   

   Aleksei Sviridkin (lexfrei)

   on Aug 7, 2026

   Contributor

   More actions

   Correcting my previous comment: I had the causality backwards. The cilium HelmRelease is reporting the failure, not producing it.

   In those runs the tenant cluster had zero Nodes registered at the 18m deadline, and the stalled resource the message names is always Deployment/cozy-cilium/cilium-operator, not the DaemonSet. The operator is a 2-replica Deployment with nowhere to schedule, so it trips progressDeadlineExceeded and kstatus renders that as Failed, which the Helm waiter treats as terminal. That is why it reports "failed early" in seconds instead of waiting out the 10m timeout, and why every retry aborts the same way. The upstream gate is the worker guest: DataVolume Succeeded, VMI Running and Ready for 15m, and the talos-csr-signer sidecar receives its first CSR at 17m30s of an 18m budget, or none at all.

   #3552 did what it claims. Across the three job logs there are no uninstall or rollback events for any cilium release, retries run every 30s matching retryInterval, and in one run a cilium release went InstallFailed to UpgradeSucceeded in place three minutes later, which under the old remediation would have been an uninstall. So the amplifier is gone; the release still reports failed for the whole window because there are no nodes to schedule on.

   What is still open is a worker-bringup question: why a Talos guest that is VMI-Ready for 15m has not reached kubelet registration. Section (b) of the diagnostics was skipped in these runs ("no tenant talosconfig on the runner"), and the capture that fixes that landed after their merge base, so the evidence that would name it was missing by construction. The branches are rebased onto current main now, so the next node-join failure carries in-guest kubelet and dmesg logs.
9. github-actions

   added

   triage/needs-triageIndicates an issue needs triage by a maintainerIndicates an issue needs triage by a maintainer

   on Aug 12, 2026
10. myasnikovdaniil commented on Sep 23, 2026

    

    myasnikovdaniil

    on Sep 23, 2026

    Contributor

    More actions

    closing this as completed by reducing nested virtualization in e2e
11. myasnikovdaniil

    closed this as completedon Sep 23, 2026