Repository navigation
talos-reconcile Job is OOMKilled at 256Mi and never completes #3519
Description
What happened
The talos-reconcile Job rendered by packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml is
limited to memory: 256Mi, and its kubectl container is OOMKilled before finishing:
pod ... Running restarts=8
lastState.terminated.exitCode = 137
restartPolicy: OnFailure with backoffLimit: 30 means it loops, and because the Job-level status stays
active, the failure is easy to miss.
Impact
The Job dies part-way through its work. In our case it reached the KamajiControlPlane patch and then died
before the TalosConfigTemplate apply, so nothing downstream of that point ever ran. Combined with the
TalosConfigTemplate immutability issue (filed separately), machine-config changes never propagate at all.
It also produces a permanently restarting pod per tenant cluster, which is noise in monitoring and a small but
constant churn against the API server.
Suggested fix
Raise the memory limit. A kubectl apply against the Cozystack API surface (many CRDs, large OpenAPI schema)
routinely needs more than 256Mi; 512Mi is a reasonable floor and the container is short-lived.
Exposing the value would also help operators who hit it on larger clusters.
Environment
Cozystack v1.6.0, 2 tenant Kubernetes clusters, Kubernetes v1.34.3 management.
Activity
dosubot commentedon Aug 3, 2026
The current memory limit of
256Mifor the kubectl container was introduced in commitb5cc2ae1(2026-06-22) and hasn't been changed since [1]. This looks like the first report of an OOM issue for this specific job — there are no prior issues or PRs tracking it.The pattern is consistent with what's been seen in other Cozystack components: PRs #2708, #2709, #2710 already bumped memory limits for Kamaji and CAPI provider controllers that were hitting the same kind of OOMKill, and etcd-operator was raised from 128Mi → 256Mi for the same reason. A
kubectl applyagainst a CRD-heavy API surface pulling down a large OpenAPI schema is a known memory-hungry operation, so 256Mi being insufficient here is plausible, especially with multiple tenant clusters.The relevant resource block is at lines ~450-456 of
talos-reconcile-job.yaml:Raising the limit to 512Mi (or making it configurable via
values.yaml) seems like a reasonable fix, in line with the adjustments already made for the other controllers.To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
fix(kubernetes): raise kubectl Job memory limits above the OOMKill floor
fix(kubernetes): raise kubectl Job memory limits above the OOMKill floor
lexfrei commentedon Sep 29, 2026
Still happens on v1.6.4. The Job template is the same as in v1.6.2, and a new Job is created whenever the render hash changes, so the 1.6.2 → 1.6.4 upgrade started it again.
I measured the same
kubectl applyof the TalosConfigTemplate from a cold cache against a management cluster with 382 CRDs: peak RSS is 282 MB with--dry-run=clientand 303 MB with--dry-run=server. So 256Mi does not fit a single attempt there.A copy of the Job with a 1Gi limit got past that step and was then rejected by the webhook from #3515. The rendered
datawas byte-identical to the live object, and the apply was still denied as an immutable spec change.mattia-eleuteri commentedon Sep 29, 2026
Fixed on
mainby #4163, which applies the TalosConfigTemplate server-side. The OOM came from the client-side merge path only: on a 438-CRD management cluster, a client-side apply against an existing object peaks at 292Mi, while the server-side apply peaks at 56Mi, well inside the existing 256Mi limit. That matches the 282–303 MB Aleksei Sviridkin (@lexfrei) measured on v1.6.4, which still uses the client-side apply. #3521, which raised the limit instead, is closed as superseded.#4163 did not carry
Fixes #3519, which is why this issue stayed open. The fix is not in any release yet, and v1.6.x, v1.6.4 included, still has the client-side apply, so it is worth a backport.Once the OOM is gone, the Job reaches the TalosConfigTemplate apply and can then hit the immutability webhook, as Aleksei Sviridkin (@lexfrei) saw with a 1Gi copy of the Job: byte-identical
data, still denied. That is #3515, addressed by #3523. Whether a server-side apply of an unchanged template passes the webhook has not been checked yet; that question belongs in #3515.