Describe the bug
The multus daemon is too small to recover a node after a CNI outage. Its CPU limit is 100m and its memory limit is 1Gi, both hardcoded in the DaemonSet template, so they can't be raised through package values. When a node comes back with a few hundred pods waiting for networking, multus can't keep up and pods never get networking back.
What I saw on a single node, after cilium-agent had been down for about two days (#4800). Once cilium was fixed, about 600 dead sandboxes and 220 pods were waiting for CNI calls. The kube-multus container was throttled in every CFS period (nr_throttled equal to nr_periods, 11690 of 11690). A DEL for a sandbox whose ADD never completed is a no-op in multus, but it waited around 190s in the queue (median), and ADD around 300s. Kubelet gives up on a CRI call after 2 minutes and retries, so the queue only grew and nothing recovered, even after restarting k3s.
After raising the CPU limit, multus processed requests in parallel and got OOM-killed against the 1Gi limit (6 restarts in a few minutes). With 2 CPU and 4Gi it cleared the backlog in about 3 minutes and the node recovered.
Environment
- Cozystack v1.7.0-alpha.3, upgraded from v1.6.0
- Provider: on-prem bare metal, single node, 128 CPUs
- Ubuntu 26.04, k3s v1.36.5, CNI chain multus → kube-ovn → cilium
To Reproduce
- Take pod networking away on a node with a few hundred pods for a while, for example with cilium-agent crash-looping.
- Bring networking back.
- Pods stay in
Unknown / ContainerCreating, kubelet logs StopPodSandbox ... DeadlineExceeded and the multus container is throttled at its CPU limit.
Expected behaviour
Multus has enough CPU and memory to work through the backlog, or the limits can be raised from package values.
Actual behaviour
The node does not recover on its own. Kubelet retries faster than multus answers.
Logs
CmdAdd (shim): timed out waiting for the condition
StopPodSandbox from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Additional context
The 100m CPU limit was already there in v1.6.0. The 1Gi memory limit came with #3495.
I'd drop the CPU limit for kube-multus, since it sits on the critical path for every pod on the node, and move its resources into values so they can be overridden.
Checklist
Describe the bug
The multus daemon is too small to recover a node after a CNI outage. Its CPU limit is 100m and its memory limit is 1Gi, both hardcoded in the DaemonSet template, so they can't be raised through package values. When a node comes back with a few hundred pods waiting for networking, multus can't keep up and pods never get networking back.
What I saw on a single node, after cilium-agent had been down for about two days (#4800). Once cilium was fixed, about 600 dead sandboxes and 220 pods were waiting for CNI calls. The
kube-multuscontainer was throttled in every CFS period (nr_throttledequal tonr_periods, 11690 of 11690). A DEL for a sandbox whose ADD never completed is a no-op in multus, but it waited around 190s in the queue (median), and ADD around 300s. Kubelet gives up on a CRI call after 2 minutes and retries, so the queue only grew and nothing recovered, even after restarting k3s.After raising the CPU limit, multus processed requests in parallel and got OOM-killed against the 1Gi limit (6 restarts in a few minutes). With 2 CPU and 4Gi it cleared the backlog in about 3 minutes and the node recovered.
Environment
To Reproduce
Unknown/ContainerCreating, kubelet logsStopPodSandbox ... DeadlineExceededand the multus container is throttled at its CPU limit.Expected behaviour
Multus has enough CPU and memory to work through the backlog, or the limits can be raised from package values.
Actual behaviour
The node does not recover on its own. Kubelet retries faster than multus answers.
Logs
Additional context
The 100m CPU limit was already there in v1.6.0. The 1Gi memory limit came with #3495.
I'd drop the CPU limit for
kube-multus, since it sits on the critical path for every pod on the node, and move its resources into values so they can be overridden.Checklist