Repository navigation
[kubernetes] Default md0 node group gets no talos-reconcile Job, so scaling it from zero stalls #3504
Description
Describe the bug
With nodeGroups left empty - the schema default {} - the kubernetes chart renders a MachineDeployment for the built-in md0 group but no talos-reconcile Job for it. That Job is the only thing that ever creates the TalosConfigTemplate the MachineDeployment references, so when the cluster-autoscaler scales md0 up from zero, each new Machine points at a bootstrap config that does not exist and never will. CAPI blocks on it indefinitely.
The cause is a single loop reading the wrong source:
- templates/cluster.yaml - the kubernetes.nodeGroups helper - substitutes the default md0 when the map is empty
- templates/talos/talos-reconcile-job.yaml - .Values.nodeGroups - the raw map, which is empty, so zero Jobs render
The chart already states the intended contract in cluster.yaml: "The MachineDeployment below still references the TCT name; CAPI's Machine controller blocks on its absence and unblocks the moment the Job applies it." For the default md0, that Job never runs.
Secondary effect: since no Job runs at all, the KamajiControlPlane.spec.network.certSANs patch that the same Job performs is also skipped.
How it appeared
Two PRs, three weeks apart:
- fix(kubernetes): make the default md0 node group removable #2936 (2026-05-27) moved the default md0 out of values.yaml into the kubernetes.nodeGroups helper, converting both raw-map loops that existed at the time (cluster.yaml, dashboard-resourcemap.yaml) to read the helper.
- feat(kubernetes): bootstrap tenant workers with Talos via CABPT #2931 (2026-06-16) added templates/talos/talos-reconcile-job.yaml as a new file with a fresh range … .Values.nodeGroups loop — written against the pattern that had been retired three weeks earlier.
This looks like an oversight rather than a deliberate choice. The file opens with a long comment documenting every other design decision (why main-phase rather than a hook, content-hash naming, ownerReference GC) and says nothing about skipping the default group; and the review pass in b5cc2ae edited that exact loop — adding a --- separator so multi-group renders stay valid YAML — without noticing which map it reads.
Why it has not been reported
The default provisions zero workers, and every e2e suite and helm-unittest fixture declares md0 explicitly, so the empty-nodeGroups render is never exercised against the Job. tests/nodegroups_default_test.yaml does render the empty case, but lists only templates/cluster.yaml, so it never looks at Jobs.
There was a window where this fired at install time rather than only on scale-up: until #3233 (2026-07-07) the MachineDeployment hardcoded replicas: 2, so an empty-nodeGroups install would have created two Machines immediately, both stuck on the missing TalosConfigTemplate. That fix returned the default group to zero replicas, pushing the bug back behind the autoscaler.
Environment
- Cozystack v1.6.0 and later (the Job was first shipped in v1.6.0)
- Any cluster installed with the default or empty nodeGroups
To Reproduce
Not observed live — this is a code-reading finding, though the render logic is unambiguous (range over an empty map yields no Jobs). To confirm:
- Create a Kubernetes CR with nodeGroups: {}, or simply omit the field.
- Deploy an unschedulable workload so the cluster-autoscaler scales md0 up, or scale the MachineDeployment manually.
- Check for a -md0 TalosConfigTemplate in the namespace.
Expected behaviour
The Machine boots and joins the cluster, because the default md0 group gets a talos-reconcile Job like any explicitly declared group.
Actual behaviour
The Machine stays pending on a TalosConfigTemplate that is never created, and no -md0 TalosConfigTemplate exists in the namespace.
Additional context
Add any other context about the problem here.
Checklist
- I have checked the documentation
- I have searched for similar issues
- I have included all required information
- I have provided clear steps to reproduce
- I have included relevant logs
Activity
dosubot commentedon Aug 1, 2026
Hi Aleksander (@keldranase)! I'm Dosu and I'm helping the Cozystack team.
This is a well-written bug report — the analysis checks out against the code.
The
kubernetes.nodeGroupshelper in_helpers.tplprovides the defaultmd0group when.Values.nodeGroupsis empty [1], andcluster.yamlcorrectly consumes it viainclude "kubernetes.nodeGroups" $ | fromYaml[2]. However,talos-reconcile-job.yamlranges directly over.Values.nodeGroups[3], so an empty map yields zero Jobs — exactly as you described.The fix is a one-line change in
talos-reconcile-job.yaml, replacing:{{- range $groupName, $group := .Values.nodeGroups }}with:
{{- range $groupName, $group := (include "kubernetes.nodeGroups" $ | fromYaml) }}This aligns the Job loop with the MachineDeployment loop so the default
md0gets its talos-reconcile Job (and the certSANs patch). You're also right that the existing test (nodegroups_default_test.yaml) only coverscluster.yamloutput and doesn't assert on the Job template [4] — extending it to includetemplates/talos/talos-reconcile-job.yamlin the empty-nodeGroups case would close the coverage gap.This is related to #3231, which fixed the hardcoded
replicas: 2that previously made this bug surface at install time rather than only on scale-up.To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
myasnikovdaniil commentedon Aug 4, 2026
Confirmed, and fixed in #3535 (targeting
main, labelledbackportso it reachesrelease-1.6).Verified by render rather than by reading: with the chart default
nodeGroups: {},templates/talos/talos-reconcile-job.yamlemits 6 documents (ServiceAccount, Role, CiliumNetworkPolicy, RoleBinding, ClusterRole, ClusterRoleBinding) and no Job; with an explicitmd0it emits 7. The fix ranges over thekubernetes.nodeGroupshelper, so the Job set tracks the MachineDeployment set. A cluster that declares its own groups renders byte-identically, andmd0stays removable.One correction to the history, because it changes which change owns the regression. The two PRs are the other way around: the Job landed on 2026-06-16 in
39cdc8866, and reading.Values.nodeGroupswas correct at that point —values.yamlstill shippednodeGroups.md0, so Helm merged the default into every render. #2936 (50748fed9, 2026-06-30) is what retired that pattern; it converted the loops incluster.yamlanddashboard-resourcemap.yamlbut not the Job's, which had existed for two weeks by then. So this is #2936 missing one of three loops rather than #2931 being written against a retired pattern. Everything else in the report holds, including the review pass inb5cc2ae1editing that exact loop without noticing which map it reads.Small addition on the install-time window: the hardcoded replicas were dropped in
6fd98dc51on 2026-07-07 andv1.6.0was tagged on 2026-07-22, so no released build ever exhibited the install-time form — only the scale-from-zero one you describe. That same commit also droppednodeGroupsfrom the two OIDC chainsaw lanes, which means those lanes do now install with an empty map; they assert that the HelmRelease exists rather than that it becomes Ready, so a worker that never boots stays invisible to them. The new suite in #3535 covers the render gap directly, and fails without the template change.Thanks for the writeup — the diagnosis was precise enough that verifying it took minutes.
fix(kubernetes): render the talos-reconcile Job for the default md0 g…
fix(kubernetes): render the talos-reconcile Job for the default md0 g…
keldranase commentedon Aug 5, 2026
myasnikovdaniil
Great, thank you for you time!
For additional context: