Skip to content

cilium-agent OOM-killed on every boot on large-memory nodes, 4Gi limit doesn't cover BPF maps #4800

Closed

Description

Describe the bug

On a node with a lot of RAM, cilium-agent can't start after a reboot on v1.7.0-alpha.3. It is OOM-killed in a loop, so the node never gets pod networking back. It is the agent's own limit, not node memory pressure, and its RSS at the kill is only ~65MB.

The limit is too small for the BPF maps the agent creates. Cilium sizes its CT, NAT and policy maps as a share of total system memory (bpf-map-dynamic-size-ratio, default 0.0025, Cozystack values don't override it). Since Linux 5.11 that map memory is charged to the memcg of the process that creates the maps. With 1.5 TiB of RAM the ratio gives roughly 3.9 GiB of maps, and the agent limit is 4Gi. The limit came with #3495 and was sized from a 575Mi peak on a 3-node dev cluster, where the maps are small.

It doesn't show up right after the upgrade. My guess is that the restarted agent reuses maps pinned in bpffs, which were created before the limit existed and are charged elsewhere. The first reboot recreates them inside the limited cgroup, and from then on the agent can't start.

Environment

  • Cozystack v1.7.0-alpha.3 (cilium chart 1.19.5), upgraded from v1.6.0
  • Provider: on-prem bare metal, single node, 128 CPUs, 1.5 TiB RAM
  • Ubuntu 26.04, kernel 7.0.0-38, k3s v1.36.5

To Reproduce

  1. Run Cozystack v1.7.0-alpha.3 on a node with roughly 1.4 TiB of RAM or more.
  2. Reboot the node.
  3. cilium-agent is OOM-killed against its cgroup limit in a loop.

Expected behaviour

The agent limit leaves room for the BPF maps the agent creates on that node.

Actual behaviour

After a reboot the node has no pod networking. Everything that needs the CNI is down, remote access agents running as pods included.

Logs

Memory cgroup out of memory: Killed process 1329390 (cilium-agent) total-vm:1437852kB, anon-rss:67380kB, file-rss:76kB, shmem-rss:0kB, UID:0 pgtables:520kB oom_score_adj:-997

The same line repeats every few minutes with a new PID.

Additional context

The 3.9 GiB is calculated from the default ratio and the node RAM, not measured. I'll add the real map memory from bpftool once the node is reachable again.

Two ways to fix it that still keep the backstop #3495 wanted. Pinning the map sizes (bpf.ctTcpMax, bpf.ctAnyMax, bpf.natMax, bpf.policyMapMax) stops map memory from growing with node RAM, so 4Gi holds everywhere. Or the limit could be derived from the map budget instead of being a fixed number.

Raising cilium.resources.limits.memory in the networking component values of the platform package should work around it. I haven't confirmed that on the node yet.

Checklist

  • I have checked the documentation
  • I have searched for similar issues
  • I have included all required information
  • I have provided clear steps to reproduce
  • I have included relevant logs

Activity

  1. added a commit that references this issue on Oct 10, 2026
    f61e13f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

kind/bugCategorizes issue or PR as related to a bugtriage/needs-triageIndicates an issue needs triage by a maintainer

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions