Repository navigation
cilium-agent OOM-killed on every boot on large-memory nodes, 4Gi limit doesn't cover BPF maps #4800
Description
Describe the bug
On a node with a lot of RAM, cilium-agent can't start after a reboot on v1.7.0-alpha.3. It is OOM-killed in a loop, so the node never gets pod networking back. It is the agent's own limit, not node memory pressure, and its RSS at the kill is only ~65MB.
The limit is too small for the BPF maps the agent creates. Cilium sizes its CT, NAT and policy maps as a share of total system memory (bpf-map-dynamic-size-ratio, default 0.0025, Cozystack values don't override it). Since Linux 5.11 that map memory is charged to the memcg of the process that creates the maps. With 1.5 TiB of RAM the ratio gives roughly 3.9 GiB of maps, and the agent limit is 4Gi. The limit came with #3495 and was sized from a 575Mi peak on a 3-node dev cluster, where the maps are small.
It doesn't show up right after the upgrade. My guess is that the restarted agent reuses maps pinned in bpffs, which were created before the limit existed and are charged elsewhere. The first reboot recreates them inside the limited cgroup, and from then on the agent can't start.
Environment
- Cozystack v1.7.0-alpha.3 (cilium chart 1.19.5), upgraded from v1.6.0
- Provider: on-prem bare metal, single node, 128 CPUs, 1.5 TiB RAM
- Ubuntu 26.04, kernel 7.0.0-38, k3s v1.36.5
To Reproduce
- Run Cozystack v1.7.0-alpha.3 on a node with roughly 1.4 TiB of RAM or more.
- Reboot the node.
- cilium-agent is OOM-killed against its cgroup limit in a loop.
Expected behaviour
The agent limit leaves room for the BPF maps the agent creates on that node.
Actual behaviour
After a reboot the node has no pod networking. Everything that needs the CNI is down, remote access agents running as pods included.
Logs
Memory cgroup out of memory: Killed process 1329390 (cilium-agent) total-vm:1437852kB, anon-rss:67380kB, file-rss:76kB, shmem-rss:0kB, UID:0 pgtables:520kB oom_score_adj:-997
The same line repeats every few minutes with a new PID.
Additional context
The 3.9 GiB is calculated from the default ratio and the node RAM, not measured. I'll add the real map memory from bpftool once the node is reachable again.
Two ways to fix it that still keep the backstop #3495 wanted. Pinning the map sizes (bpf.ctTcpMax, bpf.ctAnyMax, bpf.natMax, bpf.policyMapMax) stops map memory from growing with node RAM, so 4Gi holds everywhere. Or the limit could be derived from the map budget instead of being a fixed number.
Raising cilium.resources.limits.memory in the networking component values of the platform package should work around it. I haven't confirmed that on the node yet.
Checklist
- I have checked the documentation
- I have searched for similar issues
- I have included all required information
- I have provided clear steps to reproduce
- I have included relevant logs
Activity
fix(cilium): size the agent memory limit for its BPF maps (cozystack#…