fix(apps/kubernetes): bump tenant Cilium HR install timeout 10m -> 15m by IvanHunters · Pull Request #3230 · cozystack/cozystack

IvanHunters · 2026-07-07T12:09:43Z

Summary

The tenant Cilium HR's install timeout races the container image pull to a fresh Talos worker VM on CI runners. Bump spec.timeout on packages/apps/kubernetes/templates/helmreleases/cilium.yaml from 10m to 15m — aligned with the tenant Kubernetes install budget the rest of the stack already uses.

Root cause (from CI artifact analysis)

Reproduced on E2E run 28843794670 (job 85543970802) — the same failure signature seen on 6+ open PRs at time of writing, tracked in #3229.

Cozyreport artifact from that run, looking at snapshots/kubernetes-previous/tenantkubeconfig-test-previous-version/namespaces/cozy-cilium/v1/pod/cilium-*.yaml:

initContainerStatuses:
- name: config
  state: {waiting: {reason: PodInitializing}}
  imageID: ""                               # kubelet is still pulling
- name: mount-cgroup
  state: {waiting: {reason: PodInitializing}}
  imageID: ""
- name: apply-sysctl-overwrites
  state: {waiting: {reason: PodInitializing}}
  imageID: ""
- name: mount-bpf-fs
  state: {waiting: {reason: PodInitializing}}
  imageID: ""
- name: clean-cilium-state
  state: {waiting: {reason: PodInitializing}}
  imageID: ""

cilium-agent current.log and previous.log are zero bytes — the container never entered running phase because every init container was still waiting for iad.ocir.io/idyksih5sir9/cozystack/cilium:main@sha256:16e9a5cd3b3614571329a309fc283f5a43e7522bef91d1ff539545c0ec1bc857 to land. The HR's 10m install timeout fires, helm-controller uninstalls, retries via retries: -1, next install races the pull again.

The failure is not version-scoped:

Test	k8s minor	Wall-clock	Result
kubernetes-latest	v1.35.6	13:10	✅ (~30s under 12m `until nodes Ready` bats budget)
kubernetes-previous	v1.34.9	13:52	❌ (82s over)

Both tests sit right on the edge; whichever races the pull first wins. Cilium's own startup work (once running) completes in seconds, so 15m is pure headroom for slow first-time image pull on a runner without a warm mirror cache. Matches the 15m budget the outer tenant HR and the monitoring cozyrds annotation already use elsewhere.

Alternative fixes considered

Pre-pull Cilium image on the Talos node. Requires either a systemd unit in the Talos machineconfig or a warm-up DaemonSet — much larger change surface for the same effect.
Move the image to a closer / faster mirror. Out of scope for the tenant chart; would touch cluster-level registry config.
Bump Cilium chart. Doesn't help — the container hasn't even started; the newer chart's binary would face the same pull latency.

Test plan

Existing kubernetes-latest and kubernetes-previous E2E tests both pass on the next CI run against this branch.
Verify no downstream helm-controller behaviour change from the 5m bump (spec.timeout only controls helm install --wait deadline; interval and reconcile cadence unchanged).

Closes #3229.

Release note

fix(apps/kubernetes): raise Cilium HelmRelease install timeout to 15m so first-time image pull on a fresh Talos worker node has enough headroom on CI runners without a warm mirror cache.

Summary by CodeRabbit

Bug Fixes
- Increased the timeout for Cilium installation and upgrade reconciliation, helping long-running deployments complete more reliably.

The tenant Cilium HR's `spec.timeout: 10m` races the image pull to a fresh Talos worker VM on CI runners. Analysis of cozyreport artifact from the E2E run 28843794670 (job 85543970802) on the `kubernetes- previous` test: - Every Cilium init container on the tenant node is stuck with `state: waiting, reason: PodInitializing` and `imageID: ""` — kubelet is still fetching `iad.ocir.io/.../cozystack/cilium:main@sha256:...` when the 10m HR install timeout fires. - `cilium-agent`'s `current.log` and `previous.log` are both zero bytes — the container never entered the running phase. - On the same run, `kubernetes-latest` (v1.35.6) took 13:10 wall-clock and passed; `kubernetes-previous` (v1.34.9) took 13:52 and failed. Both minors sit right on the edge of the 10m HR budget; whichever test races the pull first wins. The delta is not version-specific. 15m matches the tenant Kubernetes install budget already used by the outer HR and by the monitoring chart's cozyrds annotation. Cilium's install work itself completes in seconds — the extra headroom is purely to absorb slow first-time image pulls on runners without a warm mirror cache. Closes #3229. Signed-off-by: IvanHunters <[email protected]>

coderabbitai · 2026-07-07T12:09:51Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7102cdc8-13f6-46b3-a0c0-cadd00c146f9

📥 Commits

Reviewing files that changed from the base of the PR and between e60ec04 and 3d60a29.

📒 Files selected for processing (1)

packages/apps/kubernetes/templates/helmreleases/cilium.yaml

📝 Walkthrough

Walkthrough

The Cilium HelmRelease reconciliation timeout in cilium.yaml is increased from 10m to 15m, accompanied by expanded explanatory comment text. No other logic or values are changed.

Changes

Cilium HelmRelease timeout update

Layer / File(s)	Summary
Increase install/upgrade timeout `packages/apps/kubernetes/templates/helmreleases/cilium.yaml`	The HelmRelease `timeout` value is changed from `10m` to `15m`, with expanded comment text explaining the change.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

cozystack/cozystack#2727: Also adjusts HelmRelease reconciliation timeouts in Kubernetes Helm chart templates, applied to csi.yaml.

Suggested reviewers: kvaps, lllamnyp, androndo, sircthulhu, myasnikovdaniil

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: increasing the tenant Cilium HelmRelease timeout from 10m to 15m.
Linked Issues check	✅ Passed	The timeout increase directly addresses the reported Cilium HelmRelease install timeout causing the tenant E2E failure.
Out of Scope Changes check	✅ Passed	Only the Cilium HelmRelease timeout was changed; no unrelated code paths or values were added.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/tenant-cilium-hr-timeout

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

gemini-code-assist · 2026-07-07T12:11:15Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request increases the HelmRelease installation timeout for Cilium to provide sufficient headroom for image pulling on cold-booted Talos worker nodes in CI environments. By aligning this timeout with the existing tenant Kubernetes install budget, the change prevents premature installation failures and ensures more reliable test execution.

Highlights

Cilium HelmRelease Timeout: Increased the install timeout for the Cilium HelmRelease from 10 minutes to 15 minutes to accommodate slow image pulls on fresh CI runner nodes.
CI Stability: Addressed intermittent E2E test failures caused by race conditions between the Helm install deadline and container image pull times, as tracked in issue E2E kubernetes-previous fails consistently: Cilium install times out inside tenant cluster on v1.34.9 nodes #3229.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	Gemini (@gemini-code-assist)	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request increases the timeout for the Cilium HelmRelease from 10m to 15m to prevent timeouts during image pulls on slow CI runners. The reviewer noted that the PR title and release-note block should use the scope kubernetes instead of apps/kubernetes to adhere to the repository's style guide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-07-07T12:11:49Z

+  # Cilium container image lives in the OCIR mirror and the tenant node
+  # has no image cache on first boot; the 10m timeout raced the pull
+  # about half the time (~13m wall-clock, see cozystack/cozystack#3229).
+  timeout: 15m


The PR title and release-note block use the scope apps/kubernetes. According to the repository style guide, the valid scope for this package is kubernetes (under the Apps category). Please update the PR title and release-note block to use kubernetes as the scope (e.g., fix(kubernetes): ...).

References

Each commit must follow Conventional Commits format with a valid scope. For the kubernetes package, the valid scope is 'kubernetes', not 'apps/kubernetes'. ^(link)

myasnikovdaniil

It is completely wrong direction of fix

myasnikovdaniil · 2026-07-07T12:14:12Z

I'll bring you issue soon

IvanHunters · 2026-07-07T12:25:04Z

Closing per myasnikovdaniil's review + #3231 analysis.

Verified against the cozyreport artifact from the failing run: the MachineDeployment.spec.replicas: 2 hardcode in packages/apps/kubernetes/templates/cluster.yaml is the actual root cause — minReplicas: 0 on the render-side OIDC test CRs is silently ignored, both kubernetes-oidc-{system,customconfig} bats churn 2× 20 GiB replicated DataVolumes each, and the subsequent kubernetes-previous sees slow LINSTOR provisioning (6+ minutes waiting on ExternalProvisioning before VMI attach in my run) which cascades into the Cilium HR --wait timeout I was trying to paper over.

Both attempts on this PR (10m → 15m spec.timeout, then disableWait: true) treat the symptom of "install runs while workers are still booting" — they don't remove the reason workers boot slowly, which is the render-side tests provisioning volumes they never intended to. The chart fix (replicas: {{ $group.minReplicas }}) + dropping nodeGroups from the render-side OIDC test CRs (both proposed in #3231) is the durable answer. Deferring to that.

No follow-up commits on this branch; safe to delete fix/tenant-cilium-hr-timeout.

myasnikovdaniil · 2026-07-07T12:25:07Z

#3229

github-actions Bot added area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug size/XS This PR changes 0-9 lines, ignoring generated files labels Jul 7, 2026

IvanHunters marked this pull request as ready for review July 7, 2026 12:11

IvanHunters requested review from Andrey Kolkov (androndo), Andrei Kvapil (kvaps), Aleksei Sviridkin (lexfrei), Timofei Larkin (lllamnyp), myasnikovdaniil and Kirill Ilin (sircthulhu) as code owners July 7, 2026 12:11

dosubot Bot added the lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor label Jul 7, 2026

gemini-code-assist Bot reviewed Jul 7, 2026

View reviewed changes

myasnikovdaniil requested changes Jul 7, 2026

View reviewed changes

IvanHunters closed this Jul 7, 2026

IvanHunters deleted the fix/tenant-cilium-hr-timeout branch July 7, 2026 12:25

Aleksei Sviridkin (lexfrei) mentioned this pull request Jul 28, 2026

E2E kubernetes-previous fails consistently: Cilium install times out inside tenant cluster on v1.34.9 nodes #3229

Closed

Navigation Menu

Repository navigation

fix(apps/kubernetes): bump tenant Cilium HR install timeout 10m -> 15m - #3230

fix(apps/kubernetes): bump tenant Cilium HR install timeout 10m -> 15m#3230
IvanHunters wants to merge 1 commit into
mainfrom
fix/tenant-cilium-hr-timeout

IvanHunters commented Jul 7, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 7, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

gemini-code-assist Bot commented Jul 7, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jul 7, 2026

Uh oh!

myasnikovdaniil left a comment

Uh oh!

myasnikovdaniil commented Jul 7, 2026

Uh oh!

IvanHunters commented Jul 7, 2026

Uh oh!

myasnikovdaniil commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

IvanHunters commented Jul 7, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root cause (from CI artifact analysis)

Alternative fixes considered

Test plan

Release note

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

gemini-code-assist Bot commented Jul 7, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

myasnikovdaniil left a comment

Choose a reason for hiding this comment

Uh oh!

myasnikovdaniil commented Jul 7, 2026

Uh oh!

IvanHunters commented Jul 7, 2026

Uh oh!

myasnikovdaniil commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

IvanHunters commented Jul 7, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 7, 2026 •

edited

Loading