Repository navigation
TalosConfigTemplate is immutable but applied on every reconcile, so machine-config changes never propagate #3515
Description
Activity
Good catch — this is a real gap. The code confirms the asymmetry you described:
KubevirtMachineTemplatealready uses content-hashed naming incluster.yaml— it computes asha256sum | trunc 6of the rendered spec and appends it to the resource name (<release>-<group>-<hash>), so any spec change creates a new object and CAPI rolls the machines [1].TalosConfigTemplate, by contrast, uses a stable name${RELEASE}-${GROUP_NAME}with no hash [2]. The Job's own comment explains this was intentional — "an unchanged render re-applies a byte-identical TCT — a CAPI no-op that never rolls existing workers" [3] — but it doesn't account for the case where the rendered content actually does change, which hits the immutable-spec webhook exactly as you describe.The
MachineDeploymentalready references one hashed template (infrastructureRef→KubevirtMachineTemplate) and one unhashed template (bootstrap.configRef→TalosConfigTemplate) [4], so adopting the same hashed-name pattern forTalosConfigTemplatewould be consistent and straightforward:- Compute a hash of the rendered
TalosConfigTemplatespec (samesha256sum | trunc 6approach). - Name it
<release>-<group>-<hash>. - Update
MachineDeployment.spec.template.spec.bootstrap.configRef.nameto include the hash. - Preserve old
TalosConfigTemplateobjects until noMachineSetreferences them (the chart already does this for oldKubevirtMachineTemplateobjects) [5].
I didn't find an existing issue or PR tracking this, so this appears to be the first report.
To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
- Compute a hash of the rendered
- added 2 commits that reference this issue
on Sep 21, 2026 - added and removed
on Sep 23, 2026 - added a commit that references this issue
on Sep 29, 2026 - added a commit that references this issue
on Sep 29, 2026 - added a commit that references this issue
on Sep 29, 2026 - added a commit that references this issue
on Sep 30, 2026
What happened
TalosConfigTemplatehas an immutable.spec(enforced by thevtalosconfigtemplate.cluster.x-k8s.iowebhook), but
packages/apps/kubernetes/templates/talos/talos-reconcile-job.yamlapplies it withkubectl applyon every reconcile. As a result, once the template exists, no change to the renderedTalos machine config can ever reach it. The apply is rejected server-side:
This is silent: the Job does not surface it as a distinct error, and the
kubernetes-*HelmRelease staysReady.Why it matters
Anything rendered into the machine config is frozen at first creation. Concretely, we hit this with
machine.network.searchDomains: our management cluster domain was misconfigured, we correctednetworking.clusterDomain, the chart then rendered the correctsearchDomains, and the template kept thestale value forever. The same applies to
certSANs,machine.registries, nameservers, and any futuremachine-config field.
The workaround is to
delete+createtheTalosConfigTemplateby hand for every tenant cluster after everyplatform upgrade, which does not scale and is easy to forget.
Reproduce
Kubernetescluster and let the workers roll (v1.6.0, Talos workers).networking.clusterDomain.kubernetes-*HelmRelease upgrade andtalos-reconcilerun.kubectl -n <tenant-ns> get talosconfigtemplate <cluster>-<group> -o yamlstill shows the old value.Suggested fix
Name the
TalosConfigTemplatewith a hash of its rendered content, the wayKubevirtMachineTemplatealready is in the same chart, and pointMachineDeployment.spec.template.spec.bootstrap.configRefat the hashed name. A content change then creates a new template instead of attempting a rejected mutation,
and CAPI rolls the machines onto it, which is the desired behaviour anyway.
Alternatively, have
talos-reconciledetect a content drift and perform the delete/create itself, but the hashedname is the idiomatic CAPI pattern and avoids a destructive step.
Environment
Cozystack v1.6.0 (packages digest
sha256:bf682087…), Talos v1.13.6 workers, Kubernetes v1.34.3 management.